Improving construct validity: Cronbach, Meehl, and Neurath's ship: Comment

Communication, Cognition and Competency Development in Healthcare: A Model for Integrating Cognitive Ethnography and Communication Skills Training in Clinical Interventions

OBJECTIVES

The aim of this study was to conduct and evaluate the Blended Learning communication skills training program. The key objective was to investigate (i) how clinical intervention studies can be designed to include cognitive, organizational, and interactive processes, and (ii) how researchers and practitioners could work with integrated methods to support the desired change.

METHODS

The method combined design and implementation of a 12-week Blended Learning communication skills training program based on the Calgary-Cambridge Guide. The training was implemented in a patient clinic at a Danish university hospital and targeted all healthcare professionals at the clinic. Cognitive ethnography was used to document and evaluate healthcare professionals' implementation and individual competency development, and support the design of in-situ simulation training scenarios.

RESULTS

Thirteen participants completed the program. The synergy within the teams, as well as the opportunities for participants to coordinate, share, discuss, and reflect on the received knowledge with a colleague or on-site researcher, affected learning positively. The knowledge transfer process was affected by negative feedback loops, such as time shortages, issues with concept development and transfer, disjuncture between the expectations of participants and instructors of the overall course structure, as well as participant insecurity and a gradual loss of motivation and compliance.

CONCLUSION

We propose a novel 3-step model for clinical interventions based on our findings and literature review. This model will effectively support the implementation of educational interventions in health care by narrowing the theory-practice gap. It will also stimulate desired change in individual behavior and organizational culture over time. Furthermore, it will work for the benefit of the clinic and may be more suitable for the implementation of communication projects than, for example, randomized setups.

Turkish Validity and Reliability Study of the Scale of Psychosocial Assessment of Candidates for Transplantation (PACT)

BACKGROUND

The Psychosocial Assessment of Candidates for Transplantation (PACT) scale shows a statistically acceptable level of reliability and validity and can guide clinicians in managing transplant candidates. This study aims to adapt PACT scale to Turkish and evaluate its validity and reliability for Turkish transplant candidates.

METHODS

This was a psychometric study of a sample of 162 patients in the organ transplant services of 2 hospitals in Turkey. The number of patients enrolled in the study was 20 times the number of items on the scale. The research data were collected using PACT. Descriptive statistics, Cronbach's alpha reliability coefficient, Pearson correlation, and factor analysis were used to evaluate the data.

RESULTS

The data were analyzed using varimax rotation in principal component analysis. The factor loadings of the items ranged from 0.56 to 0.79. The internal reliability coefficient of the scale is 0.87. It was also found that the scale accounted for the 52.82% of the total variance.

CONCLUSION

According to the results of this study, evidence of the validity and reliability of the PACT was obtained.

The Power of One Word to Paint a Halo or a Horn: Demonstrating the Halo Effect in Learner Handover and Subsequent Evaluation

PURPOSE

Learner evaluation based upon direct observation is a cornerstone of modern competency-based medical education. Learner handover has become a widely accepted practice. Cognitive attribution bias is a potential threat to the validity of learner evaluation following learner handover.

METHOD

In this 2 x 2 (learner handover: halos/horns x learner gender: man/woman) factorial, nonequivalent comparison group experimental design, registered attendees at a national medical education conference watched 1 of 2 videos (depicting a woman learner or man learner) of simulated learner-patient encounters. Data were collected on April 30 and May 1, 2022. Participants received learner handover conditioning before watching the video. The conditioning was randomized to suggest the learner they were about to watch was either "above-average" (halos) or "below-average" (horns). Following the video, participants completed an evaluation form.

RESULTS

Participants rated the learner in a video encounter preceded by a horns statement significantly lower than the learner in a video encounter preceded by a halo statement, F (1,65) = 10.15, P < .01, η 2 = .14, horns mean adj = 12.49 (CI 11.34, 13.63), halo mean adj = 15.10 (CI 13.93, 16.28). This represented a scoring difference of 2.61 points on a 30-point scale. More years of teaching experience was negatively associated with the score, F (1,65) = 13.44, P < .001, η 2 = .17.

CONCLUSIONS

Learner conditioning differing by a single word, labeling a learner as either "above-average" or "below-average" resulted in a large difference in scoring by medical educators.

Validity and reliability of the Turkish version of the Spiritual Needs Assessment Scale of Patients with Cancer

OBJECTIVE

This research was conducted to determine the psychometric characteristics of the "Spiritual Needs Assessment Scale of Patients with Cancer" in Turkish patients with cancer.

METHODS

This study used psychometric methods to test the adapted tool. The participants of this study were patients with a diagnosis of cancer in the outpatients and inpatients medical oncology and hematology clinics of a university hospital in Turkey. 400 patients determined by power analysis were included in the study. The data were collected in 2018. We used the descriptive form and "Spiritual Needs Assessment Scale of Patients with Cancer." The obtained data were evaluated using Cronbach's α reliability coefficient, Pearson's correlation coefficient, and factor analysis.

RESULTS

In the present study, the factor loads of the items were sufficient and explained 39.18% of the total variance. Cronbach's α value of the scale was 0.85.

SIGNIFICANCE OF RESULTS

As a result, it was found that the validity and reliability of the "Spiritual Needs Assessment Scale of Patients with Cancer" were ensured and they could be used in Turkish patients.

Drug Abstinence Self-Efficacy Scale (DASES): psychometric properties of the Farsi version

Background

Research has demonstrated that therapeutic interventions based on the self-efficacy theory produce positive outcomes for people who exhibit addictive behaviors, such as alcohol and drug use. Several questionnaires based on self-efficacy theory have been developed to evaluate the extent to which intervention programs can modify behavior. The present study describes the psychometric properties of the Farsi version of the Drug Abstinence Self-Efficacy Scale (DASES).

Design and methods

The forward–backward approach was employed to translate the DASES from English into Farsi. A cross-sectional study was conducted, and the psychometric properties of the Farsi version of the DASES were measured. Using a cluster sampling method, 400 male people who use drugs aged 20 years or older were selected from 10 addiction treatment clinics in Mazandaran, Iran. The internal consistency and test–retest methods were used to measure the reliability of the DASES. Face and content validity were measured, and the construct validity of the DASES was assessed through both exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). The data were analyzed using SPSS and AMOS.

Results

The results of the EFA indicated a four-factor solution for the DASES that accounted for 64.72% of the observed variance. The results obtained from the CFA demonstrated that the data fitted the model: the relative chi square (× 2/df) equaled 1.99 (p < 0.001), and the root mean square error of approximation equaled 0.071 (90% CI = 0.059–0.082). All the comparative indices of the model were equal to or greater than 0.90 (0.91, 0.93, 0.94, 0.93, and 0.90, respectively). The Cronbach’s alpha ranged from 0.90 to 0.93, proving a satisfactory reliability. Additionally, the intraclass correlation coefficient ranged from 0.75 to 0.98, which is an acceptable result.

Conclusions

This study’s results show that the Iranian version of the DASES has good psychometric properties and is appropriate for assessing substance use behaviors among Iranian addicted persons.

Applicability and cross-cultural validation of the Chinese version of the Warwick-Edinburgh mental well-being scale in patients with chronic heart failure

Background

The mental well-being of patients with chronic heart failure is likely to influence their health-related quality of life and decrease the utilization of public health resources. This study assessed the mental well-being of patients with chronic heart failure and evaluated the reliability and validity of the Warwick-Edinburgh Mental Well-Being Scale.

Methods

We conducted a cross-sectional survey from July 2016 to July 2017 among 191 patients with chronic heart failure, and examined psychometric properties of the Warwick-Edinburgh Mental Well-Being Scale, such as internal consistency, reliability, test-retest reliability, and factorial validity of the Chinese version of the Warwick-Edinburgh Mental Well-Being Scale.

Results

One-dimensional construct validity was demonstrated by confirmatory factor analysis. The psychometric properties of the Chinese version of the Warwick-Edinburgh Mental Well-Being Scale were satisfactory in our sample of patients with chronic heart failure. The internal consistency reliability was .948 and the test-retest reliability .925. The item-total correlations ranged from .405 to .872. There was a strong correlation (r = .79) between the Chinese version of the Warwick-Edinburgh Mental Well-Being Scale and the five-item World Health Organization Well-Being Index. The Chinese version of the Warwick-Edinburgh Mental Well-Being Scale appears acceptable for use in patients with chronic heart failure, and we were able to verify its reliability and validity with our sample.

Conclusions

The Chinese version of the Warwick-Edinburgh Mental Well-Being Scale is a reliable quantitative tool for evaluating mental well-being in patients with chronic heart failure in clinical settings, and this has important implications for overall assessments of mental well-being in patients with chronic heart failure.

Electronic supplementary material

The online version of this article (10.1186/s12955-019-1120-2) contains supplementary material, which is available to authorized users.

The "Quantifying Construct Validity" Procedure: Its Role, Value, Interpretations, and Computation

The current article enhances the test validation process by addressing important issues with the quantifying construct validity (QCV) procedure. The QCV procedure is intended to help researchers systematically and objectively evaluate the degree to which a pattern of convergent and discriminant validity correlations correspond to a priori hypotheses. Although the QCV procedure holds promise as a psychometric tool and has enjoyed some use, at least three factors have likely limited the frequency and accuracy of its use-questions regarding its role and utility in test validation, a lack of clarity about its key concepts, and a lack of integration with widely available statistical software. We address these important issues and provide psychometrically grounded recommendations for applying the QCV procedure. This work facilitates the understanding, computation, and useful application of the QCV procedure, and ultimately it is intended to enhance work in test validation.

Assessing the Proactive and Reactive Dimensions of Criminal Thought Process: Divergent Patterns of Correlation With Variable- and Person-Level Measures of Criminal Risk and Future Outcome

The goal of this study was to determine whether measures of proactive and reactive criminal thinking display divergent patterns of correlation with outside criteria. A sample of 3,039 male medium-security federal prisoners who completed the Psychological Inventory of Criminal Thinking Styles (PICTS) served as participants in this study. Despite being highly correlated (r = .75), the PICTS proactive and reactive scales displayed divergent patterns of correlation with the eight risk/outcome measures. As predicted, the proactive scale corresponded with lower criminal risk, older age of first conviction, and decreased odds of prior substance misuse and mental illness, whereas the reactive scale corresponded with higher criminal risk, earlier age of first conviction, greater odds of prior substance misuse and mental illness, and more evidence of subsequent arrest. Contrary to predictions, the proactive scale was associated with increased rather than decreased commission of disciplinary infractions in prison. When participants with elevated proactive scores were compared to participants with elevated reactive scores on the eight risk/outcome variables, the results revealed that the two profiles were moderately negatively correlated. Thus, although proactive criminal thinking is associated with below-average criminal risk and below-average future negative outcomes, reactive criminal thinking does just the opposite.

Work-related stress assessed by a text message single-item stress question

Background

Given the prevalence of work stress-related ill-health in the Western world, it is important to find cost-effective, easy-to-use and valid measures which can be used both in research and in practice.

Aims

To examine the validity and reliability of the single-item stress question (SISQ), distributed weekly by short message service (SMS) and used for measurement of work-related stress.

Methods

The convergent validity was assessed through associations between the SISQ and subscales of the Job Demand–Control–Support model, the Effort–Reward Imbalance model and scales measuring depression, exhaustion and sleep. The predictive validity was assessed using SISQ data collected through SMS. The reliability was analysed by the test–retest procedure.

Results

Correlations between the SISQ and all the subscales except for job strain and esteem reward were significant, ranging from −0.186 to 0.627. The SISQ could also predict sick leave, depression and exhaustion at 12-month follow-up. The analysis on reliability revealed a satisfactory stability with a weighted kappa between 0.804 and 0.868.

Conclusions

The SISQ, administered through SMS, can be used for the screening of stress levels in a working population.

Translation and validation of a Chinese version of the Warwick-Edinburgh Mental Well-being Scale with undergraduate nursing trainees

WHAT IS KNOWN ON THE SUBJECT?: Warwick-Edinburgh Mental Well-being Scale (WEMWBS) which is a measurement of mental health and well-being, was translated into Chinese to assess the reliability and validity in a sample of undergraduate nursing trainees in Wenzhou. WHAT THIS PAPER ADDS TO EXISTING KNOWLEDGE?: Currently, there is no a reliable assessment tool to systematically evaluate mental well-being in mainland China. Therefore, we undertook the translation and cultural adaptation of the WEMWBS and tested the scale's reliability and validity. The reliability and validity of the Chinese version of the Short Warwick-Edinburgh Mental Well-being Scale have been demonstrated in Hong Kong, but the WEMWBS has not been validated in mainland China in either clinical or non-clinical settings. WHAT ARE THE IMPLICATIONS FOR PRACTICE?: It might provide a reliable quantitative tool to evaluate mental well-being in non-clinical settings, which has important implications for mental health nursing. In future,it might be a reliable quantitative tool for evaluating mental well-being in other population such as hospitalized patients to improve their life quality.

ABSTRACT

Introduction There is growing interest worldwide in the evaluation of positive mental health and wellbeing. Aim This study was to translate the Warwick-Edinburgh Mental Well-being Scale (WEMWBS) into Chinese and assess the reliability and validity of this scale in a sample of undergraduate nursing trainees. Methods A forward and backward translation protocol was used to translate the scale into Chinese. The psychometric properties of the C-WEMWBS, such as internal consistency reliability, test-retest reliability and factorial validity, were analysed. Results All 14 items of the initial C-WEMWBS were rated as conceptually equivalent to those of the original version. Confirmatory factor analysis indicated the emergence of a single factor. The psychometric properties of the C-WEMWBS were satisfactory, with an internal consistency reliability of .94 and a test-retest reliability of .83. The item-total correlations ranged from .66 to .83. There was a moderate relationship (r = .52) between the C-WEMWBS and the WHO-5. Implications for practice This was the first Chinese version of the WEMWBS to be developed and psychometrically assessed with a sample of undergraduate nursing trainees. It provides a reliable quantitative tool to evaluate mental well-being in non-clinical settings, which has important implications for mental health nursing.

The validity of the multi-informant approach to assessing child and adolescent mental health

Child and adolescent patients may display mental health concerns within some contexts and not others (e.g., home vs. school). Thus, understanding the specific contexts in which patients display concerns may assist mental health professionals in tailoring treatments to patients' needs. Consequently, clinical assessments often include reports from multiple informants who vary in the contexts in which they observe patients' behavior (e.g., patients, parents, teachers). Previous meta-analyses indicate that informants' reports correlate at low-to-moderate magnitudes. However, is it valid to interpret low correspondence among reports as indicating that patients display concerns in some contexts and not others? We meta-analyzed 341 studies published between 1989 and 2014 that reported cross-informant correspondence estimates, and observed low-to-moderate correspondence (mean internalizing: r = .25; mean externalizing: r = .30; mean overall: r = .28). Informant pair, mental health domain, and measurement method moderated magnitudes of correspondence. These robust findings have informed the development of concepts for interpreting multi-informant assessments, allowing researchers to draw specific predictions about the incremental and construct validity of these assessments. In turn, we critically evaluated research on the incremental and construct validity of the multi-informant approach to clinical child and adolescent assessment. In so doing, we identify crucial gaps in knowledge for future research, and provide recommendations for "best practices" in using and interpreting multi-informant assessments in clinical work and research. This article has important implications for developing personalized approaches to clinical assessment, with the goal of informing techniques for tailoring treatments to target the specific contexts where patients display concerns. (PsycINFO Database Record

The validity of the multi-informant approach to assessing child and adolescent mental health

Child and adolescent patients may display mental health concerns within some contexts and not others (e.g., home vs. school). Thus, understanding the specific contexts in which patients display concerns may assist mental health professionals in tailoring treatments to patients' needs. Consequently, clinical assessments often include reports from multiple informants who vary in the contexts in which they observe patients' behavior (e.g., patients, parents, teachers). Previous meta-analyses indicate that informants' reports correlate at low-to-moderate magnitudes. However, is it valid to interpret low correspondence among reports as indicating that patients display concerns in some contexts and not others? We meta-analyzed 341 studies published between 1989 and 2014 that reported cross-informant correspondence estimates, and observed low-to-moderate correspondence (mean internalizing: r = .25; mean externalizing: r = .30; mean overall: r = .28). Informant pair, mental health domain, and measurement method moderated magnitudes of correspondence. These robust findings have informed the development of concepts for interpreting multi-informant assessments, allowing researchers to draw specific predictions about the incremental and construct validity of these assessments. In turn, we critically evaluated research on the incremental and construct validity of the multi-informant approach to clinical child and adolescent assessment. In so doing, we identify crucial gaps in knowledge for future research, and provide recommendations for "best practices" in using and interpreting multi-informant assessments in clinical work and research. This article has important implications for developing personalized approaches to clinical assessment, with the goal of informing techniques for tailoring treatments to target the specific contexts where patients display concerns. (PsycINFO Database Record

Construct measurement quality improves predictive accuracy in violence risk assessment: an illustration using the personality assessment inventory

Much of the risk assessment literature has focused on the predictive validity of risk assessment tools. However, these tools often comprise a list of risk factors that are themselves complex constructs, and focusing on the quality of measurement of individual risk factors may improve the predictive validity of the tools. The present study illustrates this concern using the Antisocial Features and Aggression scales of the Personality Assessment Inventory (Morey, 1991). In a sample of 1,545 prison inmates and offenders undergoing treatment for substance abuse (85% male), we evaluated (a) the factorial validity of the ANT and AGG scales, (b) the utility of original ANT and AGG scales and newly derived ANT and AGG scales for predicting antisocial outcomes (recidivism and institutional infractions), and (c) whether items with a stronger relationship to the underlying constructs (higher factor loadings) were in turn more strongly related to antisocial outcomes. Confirmatory factor analyses (CFAs) indicated that ANT and AGG items were not structured optimally in these data in terms of correspondence to the subscale structure identified in the PAI manual. Exploratory factor analyses were conducted on a random split-half of the sample to derive optimized alternative factor structures, and cross-validated in the second split-half using CFA. Four-factor models emerged for both the ANT and AGG scales, and, as predicted, the size of item factor loadings was associated with the strength with which items were associated with institutional infractions and community recidivism. This suggests that the quality by which a construct is measured is associated with its predictive strength. Implications for risk assessment are discussed.

Evidence for the criterion validity and clinical utility of the Pathological Narcissism Inventory

In this study, the authors evaluated aspects of criterion validity and clinical utility of the grandiosity and vulnerability components of the Pathological Narcissism Inventory (PNI) using two undergraduate samples (N = 299 and 500). Criterion validity was assessed by evaluating the correlations of narcissistic grandiosity and narcissistic vulnerability with established indices of normal personality traits, psychopathology and clinical concerns, and pathological personality traits. Overall, the pattern of correlations supported the convergent and discriminant validity of grandiose and vulnerable conceptualizations of pathological narcissism as measured by the PNI. Clinical utility was assessed by evaluating the extent to which clinicians without specific training in pathological narcissism as well as clinicians with expertise in pathological narcissism could accurately predict the correlates of PNI grandiosity and vulnerability with normal and pathological personality traits and psychopathology. The r(contrast-cv) coefficient provided a global index of accuracy in clinicians' predictions that was more fully elaborated by examining systematic discrepancies across groups. Overall, novice and expert clinicians were generally able to predict criterion correlations, with some exceptions (e.g., counter to predictions, pathological narcissism was negatively associated with treatment resistance). These results provide further evidence regarding the validity and utility of the narcissistic grandiosity and narcissistic vulnerability constructs as measured by the PNI.

Assessing the external correlates of alternative factor models of the Psychopathic Personality Inventory-short form across three samples

This study investigated various theoretically relevant correlates of a short form of the Psychopathic Personality Inventory (PPI; Lilienfeld & Andrews, 1996) using archival data from large undergraduate, foster care, and juvenile offender samples. External correlates of the 2 primary scales (PPI-I and PPI-II) and the Coldheartedness subscale were for the most part consistent with prior findings. Analyses for an alternate factor model in which the Fearlessness subscale loaded onto PPI-II (rather than PPI-I) resulted in relatively few substantial changes to the pattern of correlations with criterion measures, but a third factor that included the Coldheartedness and Carefree Nonplanfulness subscales functioned differently than Coldheartedness alone in these data.

The compulsive sexual behavior inventory: psychometric properties

Compulsive sexual behavior (CSB) is a putative clinical syndrome characterized by the experience of sexual urges, sexually arousing fantasies, and sexual behaviors that are recurrent, intense, and a distressful interference in one's daily life. Although the putative phenomenology of CSB has been described in the literature, the lack of a reliable, valid assessment tool has made investigation of prevalence, co-factors, and etiologic factors difficult. This study examined the further development of the Compulsive Sexual Behavior Inventory (CSBI) using a sample of 1,026 Latino men who have sex with men recruited and assessed using web-based technology. The scale showed a two factor structure (control and violence). Further, the CSBI and its subscales showed indications of validity in that those engaging in CSB-type sexual behavior (being drunk or high, feeling lonely or depressed, and feeling driven) had scores indicative of greater CSB. Those with scores above the median had more sexual partners and engaged in more unprotected anal intercourse than those with CSBI scores below the median. Additionally, the instrument showed equivalence when administered in English and Spanish.

Extending basic principles of measurement models to the design and validation of Patient Reported Outcomes

A recently published article by the Scientific Advisory Committee of the Medical Outcomes Trust presents guidelines for selecting and evaluating health status and health-related quality of life measures used in health outcomes research. In their article, they propose a number of validation and performance criteria with which to evaluate such self-report measures. We provide an alternate, yet complementary, perspective by extending the types of measurement models which are available to the instrument designer. During psychometric development or selection of a Patient Reported Outcome measure it is necessary to determine which, of the five types of measurement models, the measure is based on; 1) a Multiple Effect Indicator model, 2) a Multiple Cause Indicator model, 3) a Single Item Effect Indicator model, 4) a Single Item Cause Indicator model, or 5) a Mixed Multiple Indicator model. Specification of the measurement model has a major influence on decisions about item and scale design, the appropriate application of statistical validation methods, and the suitability of the resulting measure for a particular use in clinical and population-based outcomes research activities.

On the complexity of quantifying construct validity

To validate a measure of a construct is to validate a theory: The process includes theory specification, theory elaboration, choice of methods, choice of research design, reliance on necessary auxiliary theories, and empirical observation. Evaluating the success of the process is likely to be enhanced through quantification. To date, scientists have quantified the final step of empirical observation, but earlier steps in the validation process appear more difficult to measure. In this issue, D. Westen and R. Rosenthal (2005) emphasize that threats to validity are often methodological ones that precede empirical observation. Yet, their earlier attempt to quantify construct validity focused only on the final, empirical observation step of the process. For that reason, it can produce positive results despite real theoretical and methodological threats to validity. Nonetheless, the laudable attempt by these distinguished authors to quantify construct validity reminds us of the complexity of the construct validation process.