Estimation of IRT graded response models: limited versus full information methods

Evaluation and refinement of Self-Directed Learning Readiness Scale for medical students

PURPOSE

This study evaluated the underlying subdomain structure of the Self-Directed Learning Readiness Scale (SDLRS) for medical students and refined the instrument to measure the subdomains to provide evidence for construct validity. Developing self-directed learners is a well-recognized goal amongst medical educators. The SDLRS has been frequently used, however, lack of construct validity makes it difficult to interpret results.

METHODS

To identify the valid subdomains of the SDLRS, items were calibrated with the graded response model (GRM) and results were used to construct a 30-item short form. Short-form validity was evaluated by examining the correspondence between the total scores from the short form and the original instrument for individual students.

RESULTS

A five-subdomain model explained the SDLRS item response data reasonably well. These included: (1) initiative and independence in learning, (2) self-concept as an effective learner, (3) openness to learning opportunity, (4) love of learning, and (5) acceptance for one's own learning. The unidimensional GRM for each subdomain fits the data better than multi-dimensional models. The total scores from the refined short form and the original form were correlated at 0.98 and the mean difference was 1.33, providing evidence for validation. Nearly 91% of 179 respondents were accurately classified within the low, average, and high readiness groups.

CONCLUSION

Sufficient evidence was obtained for the validity and reliability of the refined 30-item short-form targeting five subdomains to measure medical students' readiness to engage in self-directed learning.

Detecting Differential Item Functioning in Multidimensional Graded Response Models With Recursive Partitioning

Differential item functioning (DIF) is a common challenge when examining latent traits in large scale surveys. In recent work, methods from the field of machine learning such as model-based recursive partitioning have been proposed to identify subgroups with DIF when little theoretical guidance and many potential subgroups are available. On this basis, we propose and compare recursive partitioning techniques for detecting DIF with a focus on measurement models with multiple latent variables and ordinal response data. We implement tree-based approaches for identifying subgroups that contribute to DIF in multidimensional latent variable modeling and propose a robust, yet scalable extension, inspired by random forests. The proposed techniques are applied and compared with simulations. We show that the proposed methods are able to efficiently detect DIF and allow to extract decision rules that lead to subgroups with well fitting models.

Multidimensional IRT for forced choice tests: A literature review

The Multidimensional Forced Choice (MFC) test is frequently utilized in non-cognitive evaluations because of its effectiveness in reducing response bias commonly associated with the conventional Likert scale. Nonetheless, it is critical to recognize that the MFC test generates ipsative data, a type of measurement that has been criticized due to its limited applicability for comparing individuals. Multidimensional item response theory (MIRT) models have recently sparked renewed interest among academics and professionals. This is largely due to the development of several models that make it easier to collect normative data from forced-choice tests. The paper introduces a modeling framework made up of three key components: response format, measurement model, and decision theory. Under this paradigm, four IRT models were chosen as examples. Following that, a comprehensive study is carried out to compare and characterize the parameter estimation techniques used in MFC-IRT models. This work then examines empirical research on the concept by analyzing three distinct domains: parameter invariance testing, computerized adaptive testing (CAT), and validity investigation. Finally, it is recommended that future research initiatives follow four distinct paths: modeling, parameter invariance testing, forced-choice CAT, and validity studies.

Evaluating Close Fit in Ordinal Factor Analysis Models With Multiply Imputed Data

Multiple imputation (MI) is one of the recommended techniques for handling missing data in ordinal factor analysis models. However, methods for computing MI-based fit indices under ordinal factor analysis models have yet to be developed. In this short note, we introduced the methods of using the standardized root mean squared residual (SRMR) and the root mean square error of approximation (RMSEA) to assess the fit of ordinal factor analysis models with multiply imputed data. Specifically, we described the procedure for computing the MI-based sample estimates and constructing the confidence intervals. Simulation results showed that the proposed methods could yield sufficiently accurate point and interval estimates for both SRMR and RMSEA, especially in conditions with larger sample sizes, less missing data, more response categories, and higher degrees of misfit. Based on the findings, implications and recommendations were discussed.

Development and Psychometric Evaluation of the Open-Source Challenging Behavior Scale (OS-CBS)

At present, there are no brief, freely-available, informant-report measures that evaluate key challenging behaviors relevant to youth with autism spectrum disorder (ASD) or other developmental disabilities (DD). This paper describes the development, refinement, and initial psychometric evaluation of a new 18-item measure, the Open-Source Challenging Behavior Scale (OS-CBS). In a large sample (n = 2004, 169 with ASD, ages 2-17), results of psychometric analyses indicated a clear factor structure (property destruction, aggression, elopement, conduct problems, and self-injury and a general factor with high loadings from all items) based on exploratory structural equation modeling, good scale reliability (α = .66-.83 for subscales, α = .91 total scale), measurement invariance across demographics, and good construct validity. The OS-CBS is a psychometrically-sound instrument for screening and monitoring intervention progress.

What Affects the Quality of Score Transformations? Potential Issues in True-Score Equating Using the Partial Credit Model

This simulation study investigated to what extent departures from construct similarity as well as differences in the difficulty and targeting of scales impact the score transformation when scales are equated by means of concurrent calibration using the partial credit model with a common person design. Practical implications of the simulation results are discussed with a focus on scale equating in health-related research settings. The study simulated data for two scales, varying the number of items and the sample sizes. The factor correlation between scales was used to operationalize construct similarity. Targeting of the scales was operationalized through increasing departure from equal difficulty and by varying the dispersion of the item and person parameters in each scale. The results show that low similarity between scales goes along with lower transformation precision. In cases with equal levels of similarity, precision improves in settings where the range of the item parameters is encompassing the person parameters range. With decreasing similarity, score transformation precision benefits more from good targeting. Difficulty shifts up to two logits somewhat increased the estimation bias but without affecting the transformation precision. The observed robustness against difficulty shifts supports the advantage of applying a true-score equating methods over identity equating, which was used as a naive baseline method for comparison. Finally, larger sample size did not improve the transformation precision in this study, longer scales improved only marginally the quality of the equating. The insights from the simulation study are used in a real-data example.

Using Auxiliary Item Information in the Item Parameter Estimation of a Graded Response Model for a Small to Medium Sample Size: Empirical Versus Hierarchical Bayes Estimation

Marginal maximum likelihood estimation (MMLE) is commonly used for item response theory item parameter estimation. However, sufficiently large sample sizes are not always possible when studying rare populations. In this paper, empirical Bayes and hierarchical Bayes are presented as alternatives to MMLE in small sample sizes, using auxiliary item information to estimate the item parameters of a graded response model with higher accuracy. Empirical Bayes and hierarchical Bayes methods are compared with MMLE to determine under what conditions these Bayes methods can outperform MMLE, and to determine if hierarchical Bayes can act as an acceptable alternative to MMLE in conditions where MMLE is unable to converge. In addition, empirical Bayes and hierarchical Bayes methods are compared to show how hierarchical Bayes can result in estimates of posterior variance with greater accuracy than empirical Bayes by acknowledging the uncertainty of item parameter estimates. The proposed methods were evaluated via a simulation study. Simulation results showed that hierarchical Bayes methods can be acceptable alternatives to MMLE under various testing conditions, and we provide a guideline to indicate which methods would be recommended in different research situations. R functions are provided to implement these proposed methods.

A 13-item Health of the Nation Outcome Scale (HoNOS-13): validation by item response theory (IRT) in patients with substance use disorder

BACKGROUND

The Health of the Nation Outcome Scale (HoNOS) is a widely used 12-item tool to assess mental health and social functioning. The French version has an added 13th item measuring adherence to psychotropic medication. The aim of the current study is to uncover the unknown pattern of the new item 13 and to compare the unidimensional and multidimensional fit of the new HoNOS-13 using Item Response Theory (IRT). This research question was studied among inpatients with substance use disorder (SUD).

METHODS

Six hundred and nine valid questionnaires of HoNOS-13 were analyzed using unidimensional (one-factor) and multidimensional (two-factor) IRT modeling.

RESULTS

The multidimensional model suggesting a first factor capturing psychiatric/impairment-related issues and a second factor reflecting social-related issues yielded better goodness-of-fit values compared to the unidimensional solution. This resulted in an improvement of all slope parameters which in turn translates to better discriminative power. Significant improvement in item location parameters were observed as well. The new item 13 had a good discriminative power (1.17) and covered a wide range of the latent trait (- 0.14 to 2.64).

CONCLUSIONS

We were able to validate the 13-item questionnaire including medication compliance and suggest that the HoNOS-13 can be recommended as a clinical evaluation tool to assess the problems and treatment needs for inpatients with SUD. Interestingly, the majority of item response categories are endorsed by respondents who are below and above the average levels of HoNOS. This indicates that the scale is able to discriminate between participants both at the low and at the high ends of the latent trait continuum. More importantly, the new item 13 has a good discriminative power and covers a broad range of the latent trait below and above the mean. It therefore has the desired profile of a good item and is a useful measure for the assessment of mental health and social functioning. Trial registration ClinicalTrials.gov, Identifier: NCT03551301. Registered: 11.06.2018. Retrospectively registered, https://clinicaltrials.gov/ct2/show/NCT03551301 .

The 18-item Swedish version of Ryff's psychological wellbeing scale: psychometric properties based on classical test theory and item response theory

Background

Psychological wellbeing is conceptualized as the full engagement and optimal performance in existential challenges of life. Our understanding of psychological wellbeing is important for us humans to survive, adapt, and thrive during the challenges of the 21st century. Hence, the measurement of psychological wellbeing is one cornerstone for the identification and treatment of both mental illness and health promotion. In this context, Ryff operationalized psychological wellbeing as a six-dimensional model of human characteristics: self-acceptance, positive relations with others, environmental mastery, personal growth, autonomy, and purpose in life. Ryff's Psychological Wellbeing Scale has been developed and translated into different versions. Here, we examine and describe the psychometric properties of the 18-item Swedish version of Ryff's Psychological Wellbeing Scale using both Classical Test Theory (CTT) and Item Response Theory (IRT).

Methods

The data used in the present study was earlier published elsewhere and consists of 768 participants (279 women and 489 men). In addition to the 18-item version of the scale, participants answered the Temporal Satisfaction with Life Scale, the Positive Affect Negative Affect Schedule, and the Background and Health Questionnaire. We examined, the 18-item version's factor structure using different models and its relationship with subjective wellbeing, sociodemographic factors (e.g., education level, gender, age), lifestyle habits (i.e., smoking, frequency of doing exercise, and exercise intensity), and health issues (i.e., pain and sleeping problems). We also analyzed measurement invariance with regard to gender. Moreover, as an addition to the existing literature, we analyzed the properties of the 18 items using Graded Response Model (GRM).

Results

Although the original six-factor structure showed a good fit, both CTT and IRT indicated that a five-factor model, without the purpose in life subscale, provided a better fit. The results supported the internal consistency and concurrent validity of the 18-item Swedish version. Moreover, invariance testing showed similar measurement precision by the scale across gender. Finally, we found several items, especially the purpose in life's item "I live life one day at a time and do not really think about the future," that might need revision or modification in order to improve measurement.

Conclusion

A five-factor solution is a valid and reliable measure for the assessment of psychological wellbeing in the general Swedish population. With some modifications, the scale might achieve enough accuracy to measure the more appropriate and correct six-dimensional theoretical framework as detailed by Ryff. Fortunately, Ryff's original version contains 20 items per subscale and should therefore act as a perfect pool of items in this endeavor.

A reassessment of the Resistance to Framing scale

Risky-choice and attribute framing effects are well-known cognitive biases, where choices are influenced by the way information is presented. To assess susceptibility to these framing types, the Resistance to Framing scale is often used, although its performance has rarely been extensively tested. In an online survey among university students from Bulgaria (N = 245) and North America (N = 261), we planned to examine the scale's psychometric properties, structural validity, and measurement invariance. However, some of these examinations were not possible because the scale displayed low and mostly non-significant inter-item correlations as well as low item-total correlations. Followingly, exploratory item response theory analyses indicated that the scale's reliability was low, especially for high levels of resistance to framing. This suggests problems with the scale at a basic level of conceptualization, namely that the items may not represent the same content domain. Overall, the scale in its current version is of limited use, at least in university student samples, due to the identified problems. We discuss potential remedies to these problems, as well as provide open code and data ( https://osf.io/j5n6f ) which facilitates testing the scale in other samples (e.g., general population, different languages and countries) to obtain a comprehensive picture of its performance.

The Autism Symptom Dimensions Questionnaire: Development and psychometric evaluation of a new, open-source measure of autism symptomatology

AIM

To describe the development and initial psychometric evaluation of a new, freely available measure, the Autism Symptom Dimensions Questionnaire (ASDQ).

METHOD

After development and revision of an initial 33-item version, informants completed a revised 39-item version of the ASDQ on 1467 children and adolescents (aged 2-17 years), including 104 with autism spectrum disorder (ASD).

RESULTS

The initial 33-item version of the ASDQ had good reliability and construct validity. However, only four specific symptom factors were identified, potentially due to an insufficient number of items. Factor analyses of the expanded instrument identified a general ASD factor and nine specific symptom factors with good measurement invariance across demographic groups. Scales showed good-to-excellent overall and conditional reliability. Exploratory analyses of predictive validity for ASD versus neurotypical and other developmental disability diagnoses indicated good accuracy for population and at-risk contexts.

INTERPRETATION

The ASDQ is a free and psychometrically sound informant report instrument with good reliability of measurement across a continuous range of scores and preliminary evidence of predictive validity. The measure may be a useful alternative to existing autism symptom measures but further studies with comparison of clinical diagnoses using criterion-standard instruments are needed.

WHAT THIS PAPER ADDS

The Autism Symptom Dimensions Questionnaire (ASDQ) is a new, freely available measure of autism symptoms. The ASDQ showed reliable and accurate measurement of autism symptoms. The measure had good screening efficiency for autism spectrum disorder relative to other developmental conditions.

Multidimensional Forced-Choice CAT With Dominance Items: An Empirical Comparison With Optimal Static Testing Under Different Desirability Matching

Several forced-choice (FC) computerized adaptive tests (CATs) have emerged in the field of organizational psychology, all of them employing ideal-point items. However, despite most items developed historically follow dominance response models, research on FC CAT using dominance items is limited. Existing research is heavily dominated by simulations and lacking in empirical deployment. This empirical study trialed a FC CAT with dominance items described by the Thurstonian Item Response Theory model with research participants. This study investigated important practical issues such as the implications of adaptive item selection and social desirability balancing criteria on score distributions, measurement accuracy and participant perceptions. Moreover, nonadaptive but optimal tests of similar design were trialed alongside the CATs to provide a baseline for comparison, helping to quantify the return on investment when converting an otherwise-optimized static assessment into an adaptive one. Although the benefit of adaptive item selection in improving measurement precision was confirmed, results also indicated that at shorter test lengths CAT had no notable advantage compared with optimal static tests. Taking a holistic view incorporating both psychometric and operational considerations, implications for the design and deployment of FC assessments in research and practice are discussed.

The impact of the first wave of COVID-19 on students' attainment, analysed by IRT modelling method

Universities around the world were closed for several months to slow down the spread of the COVID-19 pandemic. During this crisis, a tremendous amount of effort was made to use online education to support the teaching and learning process. The COVID-19 pandemic gave us a profound insight into how online education can radically affect students and how students adapt to new challenges. The question is how switching to online education affected dropout? This study shows the results of a research project clarifying the impact of the transition to online courses on dropouts. The data analysed are from a large public university in Europe where online education was introduced in March 2020. This study compares the academic progress of students newly enroled in 2018 and 2019 using IRT modelling. The results show that (1) this period did not contribute significantly to the increase in dropout, and we managed to retain our students.(2) Subjects became more achievable during online education, and students with less ability were also able to pass their exams. (3) Students who participated in online education reported lower average grade points than those who participated in on-campus education. Consequently, on-campus students could win better scholarships because of better grades than students who participated in online education. Analysing students' results could help (1) resolve management issues regarding scholarship problems and (2) administrators develop programmes to increase retention in online education.

Heywood Cases in Unidimensional Factor Models and Item Response Models for Binary Data

Heywood cases are known from linear factor analysis literature as variables with communalities larger than 1.00, and in present day factor models, the problem also shows in negative residual variances. For binary data, factor models for ordinal data can be applied with either delta parameterization or theta parametrization. The former is more common than the latter and can yield Heywood cases when limited information estimation is used. The same problem shows up as non convergence cases in theta parameterized factor models and as extremely large discriminations in item response theory (IRT) models. In this study, we explain why the same problem appears in different forms depending on the method of analysis. We first discuss this issue using equations and then illustrate our conclusions using a small simulation study, where all three methods, delta and theta parameterized ordinal factor models (with estimation based on polychoric correlations and thresholds) and an IRT model (with full information estimation), are used to analyze the same datasets. The results generalize across WLS, WLSMV, and ULS estimators for the factor models for ordinal data. Finally, we analyze real data with the same three approaches. The results of the simulation study and the analysis of real data confirm the theoretical conclusions.

Overfactoring in rating scale data: A comparison between factor analysis and item response theory

Educational and psychological measurement is typically based on dichotomous variables or rating scales comprising a few ordered categories. When the mean of the observed responses approaches the upper or the lower bound of the scale, the distribution of the data becomes skewed and, if a categorical factor model holds in the population, the Pearson correlation between variables is attenuated. The consequence of this correlation attenuation is that the traditional linear factor model renders an excessive number of factors. This article presents the results of a simulation study investigating the problem of overfactoring and some solutions. We compare five widely known approaches: (1) The maximum-likelihood factor analysis (FA) model for normal data, (2) the categorical factor analysis (FAC) model based on polychoric correlations and maximum likelihood (ML) estimation, (3) the FAC model estimated using a weighted least squares algorithm, (4) the mean corrected chi-square statistic by Satorra-Bentler to handle the lack of normality, and (5) the Samejima's graded response model (GRM) from item response theory (IRT). Likelihood-ratio chi-square, parallel analysis (PA), and categorical parallel analysis (CPA) are used as goodness-of-fit criteria to estimate the number of factors in the simulation study. Our results indicate that the maximum-likelihood estimation led to overfactoring in the presence of skewed variables both for the linear and categorical factor model. The Satorra-Bentler and GRM constitute the most reliable alternatives to estimate the number of factors.

A Generalized Partially Confirmatory Factor Analysis Framework with Mixed Bayesian Lasso Methods

This research extends the partially confirmatory approach to accommodate mixed types of data and missingness in a unified framework that can address a wide range of the confirmatory-exploratory continuum in factor analysis. A mix of Bayesian adaptive and covariance Lasso procedures was developed to estimate model parameters and regularize the loading structure and local dependence simultaneously. Several model variants were offered with different constraints for identification. The less-constrained variant can achieve sufficient condition for the more-powerful variant, although loading estimates associated with local dependence can be inflated. Parameter recovery was satisfactory, but the information on local dependence was partially lost with categorical data or missingness. A real-life example illustrated how the models can be used to obtain a more discernible loading pattern and to identify items that do not measure what they are supposed to measure. The proposed methodology has been implemented in the R package LAWBL.

Disentangling change across the time and true stability of employees' resilience using latent state model

There is debate within the literature about whether resilience should be considered a stable character trait or a dynamic, changeable process (state). Two widely used measures to assess resilience are the Connor-Davidson Resilience Scale (CD-RISC) and the Resilience Scale for Adults (RSA). The aim of this study was to evaluate the true stability (invariance) and change across time in resilience captured by these two measures. Using the perspective of Latent State-Trait theory, the aim was to decipher if the CD-RISC and the RSA are more trait-like or more state-like and to address whether true differences in resilience between participants increased (or decreased) across time. In this longitudinal study, UK-based employees (N = 378) completed the CD-RISC (10-item version) and the RSA (33-item version, aggregated and analyzed under six parcels) at three occasions over six months. A latent-state model and latent-state model with indicator specific residual factors were utilized. The analysis suggested that both questionnaires capture trait and state components of resilience. These results contribute to the discussion about how resilience scales are measuring change and stability, and how we define resilience as a more trait-like or state-like phenomena. The findings also highlight the issue of what resilience scales are measuring and whether resilience is a quantifiable construct.

An ultra-brief screening scale for pediatric obsessive-compulsive disorder: The OCI-CV-5

BACKGROUND

Obsessive-compulsive disorder (OCD) is an often disabling and chronic condition that is normally assessed using diagnostic interviews or lengthy self-report questionnaires. This makes routine screening in general health settings impractical, and as a result OCD is often under-(or mis-)recognized. The present study reports on the development of an ultra-brief version of the Obsessive-Compulsive Inventory-Child Version (OCI-CV) which may be administered routinely as a screener for pediatric OCD.

METHOD

A total of 489 youth diagnosed with OCD, 259 non-clinical controls, and 299 youth with other disorders completed the OCI-CV and other indices of psychopathology. Using item analyses, we extracted five items and examined the measure's factor structure, sensitivity and specificity, and convergent and discriminant validity.

RESULTS

We extracted five items that assess different dimensions of OCD (washing, checking, ordering, obsessing, neutralizing/counting), termed the OCI-CV-5. Results revealed that the measure possesses good to excellent psychometric properties, and a cutoff off (≥2) yielded optimal sensitivity and specificity.

LIMITATIONS

Participants were predominantly White. In addition, more research is needed to examine the OCI-CV-5's test-retest reliability and sensitivity to treatment.

CONCLUSIONS

The OCI-CV-5 shows promise as an ultra-brief self-report screener for identifying OCD in youth when in-depth assessment is unfeasible.

Psychometric evaluation of the Korean version of PROMIS self-efficacy for managing symptoms item bank: Item response theory

PURPOSE

To evaluate the psychometric properties of the Patient-Reported Outcomes Measurement Information System (PROMIS) self-efficacy for managing symptoms of the version 1.0 item bank in Korean.

METHODS

This study consisted of two phases: first, developing the Korean version of the item bank following the translation guidelines; and second, performing a cross-sectional study to evaluate its psychometric properties using the item response theory. This study enrolled 323 patients with type 2 diabetes mellitus between July and August 2020. Cronbach's α was used to assess the reliability of this item bank. Confirmatory factor analysis, using diagonally weighted least squares, was used to identify the assumptions of item response theory. Item parameter estimates including discrimination and thresholds were derived using the graded response model of the item response theory to reflect patient-reported outcomes as individualized responses.

RESULTS

The Korean version of the item bank demonstrated good reliability (Cronbach's α =.98) and its discrimination ranged from 1.82 to 4.93. The thresholds resulted in the establishment of a category response curve for each item. However, no overlap was observed among the category curves. Moreover, the differential item functioning was not significant for age, gender, and income variables.

CONCLUSION

The graded response model and differential item functioning provided qualitative evidence that demonstrated acceptable psychometric properties of symptom management self-efficacy among patients. This item bank is expected to provide adequate assessments of self-efficacy of symptom management for patients with a chronic disease, which can contribute to nursing research and intervention.

A Comparison of Modern and Popular Approaches to Calculating Reliability for Dichotomously Scored Items

Recent work on reliability coefficients has largely focused on continuous items, including critiques of Cronbach's alpha. Although two new model-based reliability coefficients have been proposed for dichotomous items (Dimitrov, 2003a,b; Green & Yang, 2009a), these approaches have yet to be compared to each other or other popular estimates of reliability such as omega, alpha, and the greatest lower bound. We seek computational improvements to one of these model-based reliability coefficients and, in addition, conduct initial Monte Carlo simulations to compare coefficients using dichotomous data. Our results suggest that such improvements to the model-based approach are warranted, while model-based approaches were generally superior.

Irritability and Emotional Impulsivity as Core Feature of ADHD and ODD in Children

The categorical approach of diagnosing mental disorders entails the problem of frequently occurring comorbidities, suggesting a more parsimonious structure of psychopathology. In this study, we therefore aim to assess how affective dysregulation (AD) is associated with attention-deficit/hyperactivity disorder (ADHD) and oppositional defiant disorder (ODD) in children. To assess AD in children aged 8–12 years (n = 391), we employed the parent version of a newly constructed parent rating scale. Following item reduction, we conducted exploratory and confirmatory factor analyses to establish a factorial structure of AD. One core dimension was identified, comprising irritability and emotional impulsivity, and two smaller dimensions, comprising positive emotionality and exuberance. Subsequently, we examined five different latent factor models – a unidimensional model, a first-order correlated factor model, a second-order correlated factor model, a traditional bifactor model, and a bifactor S-1 model, in which the first-order factor AD-Irritability/Emotional Impulsivity (II) was modeled as the general reference factor. A bifactor S-1 model with the a priori defined general reference domain AD-II provided the best fit to our data and was straightforward to interpret. This model showed excellent model fit and no anomalous factor loadings. This still held true, when comparing it to bifactor S-1 models with ADHD/ODD-related reference factors. Differential correlations with emotion regulation skills and the established Parent Proxy Anger Scale validate the interpretation of the different dimensions. Our results suggest that irritability/emotional impulsivity might be a common core feature of ADHD and ODD.

A comparison of methods to address item non-response when testing for differential item functioning in multidimensional patient-reported outcome measures

PURPOSE

Item non-response (i.e., missing data) may mask the detection of differential item functioning (DIF) in patient-reported outcome measures or result in biased DIF estimates. Non-response can be challenging to address in ordinal data. We investigated an unsupervised machine-learning method for ordinal item-level imputation and compared it with commonly-used item non-response methods when testing for DIF.

METHODS

Computer simulation and real-world data were used to assess several item non-response methods using the item response theory likelihood ratio test for DIF. The methods included: (a) list-wise deletion (LD), (b) half-mean imputation (HMI), (c) full information maximum likelihood (FIML), and (d) non-negative matrix factorization (NNMF), which adopts a machine-learning approach to impute missing values. Control of Type I error rates were evaluated using a liberal robustness criterion for α = 0.05 (i.e., 0.025-0.075). Statistical power was assessed with and without adoption of an item non-response method; differences > 10% were considered substantial.

RESULTS

Type I error rates for detecting DIF using LD, FIML and NNMF methods were controlled within the bounds of the robustness criterion for > 95% of simulation conditions, although the NNMF occasionally resulted in inflated rates. The HMI method always resulted in inflated error rates with 50% missing data. Differences in power to detect moderate DIF effects for LD, FIML and NNMF methods were substantial with 50% missing data and otherwise insubstantial.

CONCLUSION

The NNMF method demonstrated comparable performance to commonly-used non-response methods. This computationally-efficient method represents a promising approach to address item-level non-response when testing for DIF.

The Impact of Item Feature and Response Preference in a Mixed-Format Design

A combination of positively and negatively worded items (termed a mixed-format design) has been widely adopted in personality and attitude assessments. While advocates claim that the inclusion of positively and negatively worded items will encourage respondents to process the items more carefully and avoid response preference, others have reported that negatively worded (NW) items may induce a nuisance factor and contaminate scale scores. The present study examined the extent of the impact of the NW-item feature and further investigated whether a mixed-format design could effectively control acquiescence and the preference for extreme response options using two datasets (Attitude toward Peace Walls, and International Personality Item Pool). A proposed multidimensional item response model was implemented to simultaneously estimate the impact of item feature and response preference. The results suggested that NW items induced an impact on item responses and that affirmative preference was negligible, regardless of the proportion of NW items in a scale. However, participants' extremity preference was large in both balanced and imbalanced mixed-format designs. It concludes that the impact of the NW-item feature is not negligible in a mixed-format scale, which exhibits good control of acquiescence but not extremity preference.

Pairwise likelihood estimation for confirmatory factor analysis models with categorical variables and data that are missing at random

Methods for the treatment of item non-response in attitudinal scales and in large-scale assessments under the pairwise likelihood (PL) estimation framework and under a missing at random (MAR) mechanism are proposed. Under a full information likelihood estimation framework and MAR, ignorability of the missing data mechanism does not lead to biased estimates. However, this is not the case for pseudo-likelihood approaches such as the PL. We develop and study the performance of three strategies for incorporating missing values into confirmatory factor analysis under the PL framework, the complete-pairs (CP), the available-cases (AC) and the doubly robust (DR) approaches. The CP and AC require only a model for the observed data and standard errors are easy to compute. Doubly-robust versions of the PL estimation require a predictive model for the missing responses given the observed ones and are computationally more demanding than the AC and CP. A simulation study is used to compare the proposed methods. The proposed methods are employed to analyze the UK data on numeracy and literacy collected as part of the OECD Survey of Adult Skills.

Robust and Nonrobust Linking of Two Groups for the Rasch Model with Balanced and Unbalanced Random DIF: A Comparative Simulation Study and the Simultaneous Assessment of Standard Errors and Linking Errors with Resampling Techniques

In this article, the Rasch model is used for assessing a mean difference between two groups for a test of dichotomous items. It is assumed that random differential item functioning (DIF) exists that can bias group differences. The case of balanced DIF is distinguished from the case of unbalanced DIF. In balanced DIF, DIF effects on average cancel out. In contrast, in unbalanced DIF, the expected value of DIF effects can differ from zero and on average favor a particular group. Robust linking methods (e.g., invariance alignment) aim at determining group mean differences that are robust to the presence of DIF. In contrast, group differences obtained from nonrobust linking methods (e.g., Haebara linking) can be affected by the presence of a few DIF effects. Alternative robust and nonrobust linking methods are compared in a simulation study under various simulation conditions. It turned out that robust linking methods are preferred over nonrobust alternatives in the case of unbalanced DIF effects. Moreover, the theory of M-estimation, as an important approach to robust statistical estimation suitable for data with asymmetric errors, is used to study the asymptotic behavior of linking estimators if the number of items tends to infinity. These results give insights into the asymptotic bias and the estimation of linking errors that represent the variability in estimates due to selecting items in a test. Moreover, M-estimation is also used in an analytical treatment to assess standard errors and linking errors simultaneously. Finally, double jackknife and double half sampling methods are introduced and evaluated in a simulation study to assess standard errors and linking errors simultaneously. Half sampling outperformed jackknife estimators for the assessment of variability of estimates from robust linking methods.

Efficient and precise Ultra-QuickDASH scale measuring lymphedema impact developed using computerized adaptive testing

Purpose

This study aimed to evaluate and improve the accuracy and efficiency of the QuickDASH for use in assessment of limb function in patients with upper extremity lymphedema using modern psychometric techniques.

Method

We conducted confirmative factor analysis (CFA) and Mokken analysis to examine the assumption of unidimensionality for IRT model on data from 285 patients who completed the QuickDASH, and then fit the data to Samejima’s graded response model (GRM) and assessed the assumption of local independence of items and calibrated the item responses for CAT simulation.

Results

Initial CFA and Mokken analyses demonstrated good scalability of items and unidimensionality. However, the local independence of items assumption was violated between items 9 (severity of pain) and 11 (sleeping difficulty due to pain) (Yen’s Q3 = 0.46) and disordered thresholds were evident for item 5 (cutting food). After addressing these breaches of assumptions, the re-analyzed GRM with the remaining 10 items achieved an improved fit. Simulation of CAT administration demonstrated a high correlation between scores on the CAT and the QuickDash (r = 0.98). Items 2 (doing heavy chores) and 8 (limiting work or daily activities) were the most frequently used. The correlation among factor scores derived from the QuickDASH version with 11 items and the Ultra-QuickDASH version with items 2 and 8 was as high as 0.91.

Conclusion

By administering just these two best performing QuickDash items we can obtain estimates that are very similar to those obtained from the full-length QuickDash without the need for CAT technology.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11136-021-02979-y.

Assessing the Accuracy of Errors of Measurement. Implications for Assessing Reliable Change in Clinical settings

Item response theory (IRT) models are non-linear latent variable models for discrete measures, whereas factor analysis (FA) is a latent variable model for continuous measures. In FA, the standard error (SE) of individuals' scores is common for all individuals. In IRT, the SE depends on the individual's score, and the SE function is to be provided. The empirical standard deviation of the scores across discrete ranges should also be computed to inform the extent to which IRT SEs overestimate or underestimate the variability of the scores. Within the target range of scores the test was designed to measure, one should expect IRT SEs to be smaller and more precise than FA SEs, and therefore preferable to assess clinical change. Outside the target range, IRT SEs may be too large and more imprecise than FA SEs, and FA more precise to assess change. As a result, whether FA or IRT characterize reliable change more accurately in a sample will depend on the proportion of individuals within or outside the IRT target score range. An application is provided to illustrate these concepts.

Psychometric properties of the PROMIS-57 questionnaire, Norwegian version

Purpose

The aims of this cross-sectional study were to explore reliability and validity of the Norwegian version of the Patient-Reported Outcome Measurement System^®—Profile 57 (PROMIS-57) questionnaire in a general population sample, n = 408, and to examine Item Response properties and factor structure.

Methods

Reliability measures were obtained from factor analysis and item response theory (IRT) methods. Correlations between PROMIS-57 and RAND-36-item health survey (RAND36) were examined for concurrent and discriminant validity. Factor structure and IRT assumptions were examined with factor analysis methods. IRT Item and model fit and graphic plots were inspected, and differential item functioning (DIF) for language, age, gender, and education level were examined.

Results

PROMIS-57 demonstrated excellent reliability and satisfactory concurrent and discriminant validity. Factor structure of seven domains was supported. IRT assumptions were met for unidimensionality, local independence, monotonicity, and invariance with no DIF of consequence for language or age groups. Estimated common variance (ECV) per domain and confirmatory factor analysis (CFA) model fit supported unidimensionality for all seven domains. The GRM IRT Model demonstrates acceptable model fit.

Conclusions

The psychometric properties and factor structure of Norwegian PROMIS-57 were satisfactory. Hence, the 57-item questionnaire along with PROMIS-29, and the corresponding 8 and 4 item short forms for physical function, anxiety, depression, fatigue, sleep disturbance, social participation ability and pain interference, are considered suitable for use in research and clinical care in Norwegian populations. Further studies on longitudinal reliability and sensitivity in patient populations and for Norwegian item calibration and/or reference scores are needed.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11136-021-02906-1.

Incremental Model Fit Assessment in the Case of Categorical Data: Tucker-Lewis Index for Item Response Theory Modeling

The Tucker-Lewis index (TLI; Tucker & Lewis, 1973), also known as the non-normed fit index (NNFI; Bentler & Bonett, 1980), is one of the numerous incremental fit indices widely used in linear mean and covariance structure modeling, particularly in exploratory factor analysis, tools popular in prevention research. It augments information provided by other indices such as the root-mean-square error of approximation (RMSEA). In this paper, we develop and examine an analogous index for categorical item level data modeled with item response theory (IRT). The proposed Tucker-Lewis index for IRT (TLIRT) is based on Maydeu-Olivares and Joe's (2005) [Formula: see text] family of limited-information overall model fit statistics. The limited-information fit statistics have significantly better Chi-square approximation and power than traditional full-information Pearson or likelihood ratio statistics under realistic situations. Building on the incremental fit assessment principle, the TLIRT compares the fit of model under consideration along a spectrum of worst to best possible model fit scenarios. We examine the performance of the new index using simulated and empirical data. Results from a simulation study suggest that the new index behaves as theoretically expected, and it can offer additional insights about model fit not available from other sources. In addition, a more stringent cutoff value is perhaps needed than Hu and Bentler's (1999) traditional cutoff criterion with continuous variables. In the empirical data analysis, we use a data set from a measurement development project in support of cigarette smoking cessation research to illustrate the usefulness of the TLIRT. We noticed that had we only utilized the RMSEA index, we could have arrived at qualitatively different conclusions about model fit, depending on the choice of test statistics, an issue to which the TLIRT is relatively more immune.

Statistical estimation of structural equation models with a mixture of continuous and categorical observed variables

In the social and behavioral sciences, observed variables of mixed scale types (i.e., both continuous and categorical observed variables) have long been included in structural equation models. However, little is known about the impact of mixed continuous and categorical observed variables on the performance of existing estimation methods. This study compares two popular estimation methods with robust corrections, robust maximum likelihood (MLR) and diagonally weighted least squares (DWLS), when mixed continuous and categorical observed data are analyzed, evaluating the behavior of DWLS and MLR estimates in both measurement and full structural equation models. Monte Carlo simulation was carried out to examine the performance of DWLS and MLR in estimating model parameters, standard errors, and chi-square statistics. Two population models, a correlated three-factor measurement model and a five-factor structural equation model, were tested in combination with 36 other experimental conditions characterized by the number of observed variables' categories (2, 3, 4, 5, 6, and 7), categorical observed distribution shape (symmetry and slight asymmetry), and sample size (200, 500, and 1000). Data generation and analysis were performed with Mplus 8. Results reveal that (1) DWLS yields more accurate factor loading estimates for categorical observed variables than MLR, whereas DWLS and MLR produce comparable factor loading estimates for continuous observed variables; (2) inter-factor correlations and structural paths are estimated equally well by DWLS and MLR in nearly all conditions; (3) robust standard errors of parameter estimates obtained by MLR are slightly more accurate than those produced by DWLS in almost every condition, but the superiority of MLR over DWLS is not clearly evident once a medium or large sample is used (i.e., n = 500 or 1000); and (4) DWLS is systematically superior to MLR in controlling Type I error rates, but this superiority is attenuated with increasing sample size. The article concludes with a general discussion of the findings and some recommendations for practice and future research.

Disentangling trait, occasion-specific, and accumulated situational effects of psychological distress in adulthood: evidence from the 1958 and 1970 British birth cohorts

BACKGROUND

The trajectories of psychological distress differ between individuals, but these differences can be difficult to understand because the measures contain both consistent and situational features; however, in longitudinal studies these sources of information can be disentangled. In addition to occasion-specific features, interindividual differences can be decomposed into two sources of information: trait and carry-over effects between neighboring occasions that are not related to the trait (i.e. accumulated situational effects).

METHODS

To disentangle these three sources of variance throughout adulthood, the consistency (trait and accumulated situational effects) and occasion specificity of nine indicators of psychological distress from the Malaise Inventory were examined in two birth cohorts, the 1958 National Child Development Study (NCDS58), and the 1970 British Cohort Study (BCS70).

RESULTS

The scale was administered at ages 23, 33, 42, and 50 in NCDS58 (n = 7147), and at ages 26, 30, 34, and 42 in BCS70 (n = 6859). For each psychological symptom, more variance was consistent than occasion-specific. The majority of the consistency was due to trait variance as opposed to accumulated situational effects, indicating that an individual predisposed to be distressed at the beginning of the study remained more likely to be distressed over the whole period. Symptoms of rage were notably more consistent among males than females in both cohorts (78.1% and 81.3% variance explained by trait in NCDS58 and BCS70, respectively), and among females in the NCDS58 (69%).

CONCLUSIONS

Symptoms of psychological distress exhibited high stability throughout adulthood, especially among men, due mostly to interindividual trait differences.

Validation of the Ankylosing Spondylitis Quality of Life assessment tool in patients with non-radiographic axial spondyloarthritis

PURPOSE

To evaluate the psychometric performance of the Ankylosing Spondylitis Quality of Life (ASQoL) scale in patients with non-radiographic axial spondyloarthritis (nr-axSpA) to assess its appropriateness as an outcome measure in future clinical studies.

METHODS

Patients with active axSpA from a Phase III, randomized, double-blind, placebo-controlled trial (RAPID-axSpA, NCT01087762) were included (N = 325). Modified New York (mNY) classification criteria were used to classify patients as having ankylosing spondylitis or nr-axSpA; those with nr-axSpA were further categorized based on objective signs of inflammation. Psychometric properties of the ASQoL were assessed/documented using a mixture of modern psychometric methods and classical test theory methods. These included exploratory factor analysis and item response theory models to assess the domain structure, test the utility of a single domain relative to subdomains, assess bias, and generate statistics to guide an empirical scoring algorithm. The reliability and validity of scores were evaluated via internal consistency, test-retest reliability, concurrent validity, and known-groups validity. Score responsiveness was assessed via anchor-based clinically meaningful change, supplemented with empirical cumulative distribution function visualizations.

RESULTS

The ASQoL data were defined by four domains. However, a four-domain solution was found to be inferior to a bifactor solution in which the four domains were included within a total domain. Scoring statistics supported a unit-weighted total score. Within the nr-axSpA population with objective signs of inflammation, the ASQoL mean score had adequate reliability, validity, and ability to detect clinically meaningful change.

CONCLUSIONS

Our findings suggest that the ASQoL is an appropriate outcome measure in interventional clinical trials in patients with nr-axSpA.

Comparison of different scoring methods based on latent variable models of the PHQ-9: an individual participant data meta-analysis

BACKGROUND

Previous research on the depression scale of the Patient Health Questionnaire (PHQ-9) has found that different latent factor models have maximized empirical measures of goodness-of-fit. The clinical relevance of these differences is unclear. We aimed to investigate whether depression screening accuracy may be improved by employing latent factor model-based scoring rather than sum scores.

METHODS

We used an individual participant data meta-analysis (IPDMA) database compiled to assess the screening accuracy of the PHQ-9. We included studies that used the Structured Clinical Interview for DSM (SCID) as a reference standard and split those into calibration and validation datasets. In the calibration dataset, we estimated unidimensional, two-dimensional (separating cognitive/affective and somatic symptoms of depression), and bi-factor models, and the respective cut-offs to maximize combined sensitivity and specificity. In the validation dataset, we assessed the differences in (combined) sensitivity and specificity between the latent variable approaches and the optimal sum score (⩾10), using bootstrapping to estimate 95% confidence intervals for the differences.

RESULTS

The calibration dataset included 24 studies (4378 participants, 652 major depression cases); the validation dataset 17 studies (4252 participants, 568 cases). In the validation dataset, optimal cut-offs of the unidimensional, two-dimensional, and bi-factor models had higher sensitivity (by 0.036, 0.050, 0.049 points, respectively) but lower specificity (0.017, 0.026, 0.019, respectively) compared to the sum score cut-off of ⩾10.

CONCLUSIONS

In a comprehensive dataset of diagnostic studies, scoring using complex latent variable models do not improve screening accuracy of the PHQ-9 meaningfully as compared to the simple sum score approach.

Psychometric Properties of the Pediatric Patient-Reported Outcomes Measurement Information System Item Banks in a Dutch Clinical Sample of Children With Juvenile Idiopathic Arthritis

Objective

To assess the psychometric properties of 8 pediatric Patient‐Reported Outcomes Measurement Information System (PROMIS) item banks in a clinical sample of children with juvenile idiopathic arthritis (JIA).

Methods

A total of 154 Dutch children (mean ± SD age 14.4 ± 3.0 years; range 8–18 years) with JIA completed 8 pediatric version 1.0 PROMIS item banks (anger, anxiety, depressive symptoms, fatigue, pain interference, peer relationships, physical function mobility, physical function upper extremity) twice and the Pediatric Quality of Life Inventory (PedsQL) and the Childhood Health Assessment Questionnaire (C‐HAQ) once. Structural validity of the item banks was assessed by fitting a graded response model (GRM) and inspecting GRM fit (comparative fit index [CFI], Tucker‐Lewis index [TLI], and root mean square error of approximation [RMSEA]) and item fit (S‐X² statistic). Convergent validity (with PedsQL/C‐HAQ subdomains) and discriminative validity (active/inactive disease) were assessed. Reliability of the item banks, short forms, and computerized adaptive testing (CAT) was expressed as the SE of theta (SE[θ]). Test–retest reliability was assessed using intraclass correlation coefficients (ICCs) and smallest detectable change.

Results

All item banks had sufficient overall GRM fit (CFI >0.95, TLI >0.95, RMSEA <0.08) and no item misfit (all S‐X²P > 0.001). High correlations (>0.70) were found between most PROMIS T scores and hypothesized PedsQL/C‐HAQ (sub)domains. Mobility, pain interference, and upper extremity item banks were able to discriminate between patients with active and inactive disease. Regarding reliability, PROMIS item banks outperformed legacy instruments. Post hoc CAT simulations outperformed short forms. Test–retest reliability was strong (ICC >0.70) for all full‐length item banks and short forms, except for the peer relationships item bank.

Conclusion

The pediatric PROMIS item banks displayed sufficient psychometric properties for Dutch children with JIA. PROMIS item banks are ready for use in clinical research and practice for children with JIA.

Measuring positive psychosocial sequelae in patients with advanced cancer

OBJECTIVE

Posttraumatic growth and benefit finding describe the potential for positive changes resulting from traumatic experiences, including cancer. In oncology, these constructs are increasingly examined concurrently using the Posttraumatic Growth Inventory (PTGI) and the Benefit Finding Scale (BFS). However, distinctions between these constructs and their corresponding scales are not altogether clear, and the burden of administering 2 lengthy questionnaires is evident, particularly for patients at end-of-life.

METHOD

Baseline data from 209 participants enrolled in a randomized controlled trial evaluating the efficacy of a psychosocial intervention were analyzed. We assessed the structure and covariance of all PTGI and BFS items using item response theory to determine the extent to which these measures overlap and the potential value of their concurrent administration in patients with advanced cancer.

RESULTS

Despite conceptual differences in posttraumatic growth and benefit finding, results indicated that these measures address the same underlying construct. We subsequently analyzed 3 abbreviated scales (7, 11, and 16 items) that combine items from both scales to identify an optimal briefer combined scale. Results supported all 3 versions, with the 7- and 16-item measures appearing to have the best balance of content and concurrent validity and the 11-item version optimizing information gained with brevity.

CONCLUSIONS

These findings indicate that concurrent administration of the PTGI and BFS may be unnecessary given the high degree of overlap between these 2 measures and that a brief subset of items may adequately evaluate positive change among patients with advanced cancer while reducing participant burden. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Item Response Theory Analysis of the MDS-UPDRS Motor Examination: Tremor vs. Nontremor Items

BACKGROUND

In PD, tremor severity behaves differently from other core motor features. However, the most commonly used assessment of overall motor severity, total MDS-UPDRS Motor Examination (Part 3) score, does not account for this distinction.

OBJECTIVES

To investigate the Motor Examination (Part 3) using Item Response Theory approaches focusing on sample-independent strategies that assess how well items measure latent models of PD motor severity.

METHODS

Data from 6,298 PD patients were analyzed with graded response model Item Response Theory approaches involving two analyses all 33 Part 3 items versus the 10 tremor items and 23 bradykinesia, rigidity, gait, and posture items considered separately. The strength of relationship between items and the latent measure of parkinsonian motor severity (discrimination parameter) and calculated thresholds (location parameters) were assessed using the mirt program implemented in R (R Foundation for Statistical Computing, Vienna, Austria).

RESULTS

Analyzing all Part 3 items together, nontremor items demonstrated good discrimination parameters (mean = 1.83 ± 0.37) and range of thresholds (-1.73 to +4.42), but tremor items had poor discrimination (mean = 0.52 ± 0.76) and thresholds (-0.69 to 14.29). Segregating nontremor from tremor items in two independent analyses provided markedly improved discrimination and location parameters for both.

CONCLUSIONS

MDS-UPDRS Part 3 tremor and nontremor items have very different relations to the construct of PD severity. Strongly improved clinimetric properties for Part 3 are obtained when tremor and nontremor items are considered separately. We suggest that evaluating PD motor severity, as an operationalized summary measure, is best attained through separate analyses with tremor and nontremor motor scores. © 2020 International Parkinson and Movement Disorder Society.

A Partially Confirmatory Approach to the Multidimensional Item Response Theory with the Bayesian Lasso

For test development in the setting of multidimensional item response theory, the exploratory and confirmatory approaches lie on two ends of a continuum in terms of the loading and residual structures. Inspired by the recent development of the Bayesian Lasso (least absolute shrinkage and selection operator), this research proposes a partially confirmatory approach to estimate both structures using Bayesian regression and a covariance Lasso within a unified framework. The Bayesian hierarchical formulation is implemented using Markov chain Monte Carlo estimation, and the shrinkage parameters are estimated simultaneously. The proposed approach with different model variants and constraints was found to be flexible in addressing loading selection and local dependence. Both simulated and real-life data were analyzed to evaluate the performance of the proposed model across different situations.

Cross-cultural adaptation, factor structure, and evidence of reliability of the Smartphone Addiction Scale in Brazil

Objective:

To translate, adapt, and examine the factor structure and internal consistency of a Brazilian Portuguese version of the Smartphone Addiction Scale (SAS-BR) among university students.

Methods:

The SAS was translated and adapted for use with Brazilian samples. The resulting instrument (SAS-BR) was then administered to 356 college students. Parallel analysis was used to explore the maximum number of dimensions that underlie the assessment, and data modeling was undertaken using exploratory multidimensional item response theory (E-MIRT). The reliability of the SAS-BR was tested by computing McDonald’s omega (ω) and Cronbach’s alpha (α) coefficients.

Results:

Parallel analysis revealed a maximum of six factors. E-MIRT analysis revealed that a five-factor model was the best fit for the data. The five emergent factors were salience, positive anticipation, cyberspace-oriented relationship, preoccupation with smartphone, and physical symptoms; these factors together explained 50% of the variance. Confirmatory factor analysis, conducted using data from a second sample, replicated the five-factor structure. The internal consistency of the scale was found to be strong.

Conclusion:

The emergent factor structure of the SAS-BR was found to be similar to that of previous adaptations of this instrument for other Western countries.

The use of items and item parcels in nonlinear structural equation models

Nonlinear structural equation models within the frequentist framework were developed to work with continuous items. Applied researchers who usually work with Likert-type items choose between two strategies to estimate such models: treat items as continuous variables or create item parcels. Two Monte Carlo studies were conducted to evaluate the effects of each strategy on estimates and Type I errors for models with interaction and quadratic effects estimated using LMS. The first study evaluated the effect of asymmetry type and item quantity. The second assessed the use of item parcels and parcel configuration under equivalent conditions. Results reveal that treating items as continuous variables is not problematic when item categories are symmetrical or have opposite-direction asymmetries; however, meaningful parameter bias and increased Type I errors are produced in the case of same-direction asymmetry. Use of parcels does not overcome these problems. The results are discussed to provide recommendations for applied researchers.

Fitting Ordinal Factor Analysis Models With Missing Data: A Comparison Between Pairwise Deletion and Multiple Imputation

This study compares two missing data procedures in the context of ordinal factor analysis models: pairwise deletion (PD; the default setting in Mplus) and multiple imputation (MI). We examine which procedure demonstrates parameter estimates and model fit indices closer to those of complete data. The performance of PD and MI are compared under a wide range of conditions, including number of response categories, sample size, percent of missingness, and degree of model misfit. Results indicate that both PD and MI yield parameter estimates similar to those from analysis of complete data under conditions where the data are missing completely at random (MCAR). When the data are missing at random (MAR), PD parameter estimates are shown to be severely biased across parameter combinations in the study. When the percentage of missingness is less than 50%, MI yields parameter estimates that are similar to results from complete data. However, the fit indices (i.e., χ2, RMSEA, and WRMR) yield estimates that suggested a worse fit than results observed in complete data. We recommend that applied researchers use MI when fitting ordinal factor models with missing data. We further recommend interpreting model fit based on the TLI and CFI incremental fit indices.

Differential item functioning analysis of the CUDIT and relations with alcohol and tobacco use among men across five ethnic groups: The HELIUS study

Cannabis use has been rising despite recognition of the negative consequences associated with heavy use. The severity of these consequences has been shown to differ across racial/ethnic groups, even when controlling for consumption levels. The present study conducted an item response theory (IRT) analysis of the Cannabis Use Disorders Identification Test (CUDIT) to better understand the patterns of problematic cannabis use and their relation with other substance use across ethnic groups in the Healthy Life in an Urban Setting (HELIUS) study. CUDIT responses from 1,960 cannabis-using African Surinamese, South-Asian Surinamese, Dutch, Moroccan, and Turkish ethnic origin participants were used to test for differential item functioning (DIF) within an IRT framework. After restricting the sample to men because of low frequency of use among women, several instances of uniform DIF were identified. Multiple-group IRT analysis yielded a harmonized cannabis use phenotype that was used to estimate ethnic group differences in problematic cannabis use and its relation to alcohol and tobacco co-use. These analyses suggested that cannabis users from certain ethnic minority groups experienced higher rates of problematic use than the majority group despite lower rates of cannabis use. Further, cannabis and tobacco use were positively related across groups, whereas only ethnic minority groups showed a positive relation between cannabis and alcohol use. These results demonstrate the importance of accounting for DIF when examining group differences in problematic cannabis use, and support prior evidence suggesting that certain ethnic minority groups may be more likely to experience problematic cannabis use and alcohol co-use relative to the majority group. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Multilevel Reliability Measures of Latent Scores Within an Item Response Theory Framework

This paper evaluated multilevel reliability measures in two-level nested designs (e.g., students nested within teachers) within an item response theory framework. A simulation study was implemented to investigate the behavior of the multilevel reliability measures and the uncertainty associated with the measures in various multilevel designs regarding the number of clusters, cluster sizes, and intraclass correlations (ICCs), and in different test lengths, for two parameterizations of multilevel item response models with separate item discriminations or the same item discrimination over levels. Marginal maximum likelihood estimation (MMLE)-multiple imputation and Bayesian analysis were employed to evaluate the accuracy of the multilevel reliability measures and the empirical coverage rates of Monte Carlo (MC) confidence or credible intervals. Considering the accuracy of the multilevel reliability measures and the empirical coverage rate of the intervals, the results lead us to generally recommend MMLE-multiple imputation. In the model with separate item discriminations over levels, marginally acceptable accuracy of the multilevel reliability measures and empirical coverage rate of the MC confidence intervals were found in a limited condition, 200 clusters, 30 cluster size, .2 ICC, and 40 items, in MMLE-multiple imputation. In the model with the same item discrimination over levels, the accuracy of the multilevel reliability measures and the empirical coverage rate of the MC confidence intervals were acceptable in all multilevel designs we considered with 40 items under MMLE-multiple imputation. We discuss these findings and provide guidelines for reporting multilevel reliability measures.

Measuring and Predicting Individual Differences in Executive Functions at 14 Months: A Longitudinal Study

This study of 195 (108 boys) children seen twice during infancy (Time 1: 4.12 months; Time 2: 14.42 months) aimed to investigate the associations between and infant predictors of executive function (EF) at 14 months. Infants showed high levels of compliance with the EF tasks at 14 months. There was little evidence of cohesion among EF tasks but simple response inhibition was related to performance on two other EF tasks. Infant attention (but not parent-rated temperament) at 4 months predicted performance on two of the four EF tasks at 14 months. Results suggest that EF skills build on simpler component skills such as attention and response inhibition.

Modeling Interactions Between Latent Variables in Research on Type D Personality: A Monte Carlo Simulation and Clinical Study of Depression and Anxiety

Several approaches exist to model interactions between latent variables. However, it is unclear how these perform when item scores are skewed and ordinal. Research on Type D personality serves as a good case study for that matter. In Study 1, we fitted a multivariate interaction model to predict depression and anxiety with Type D personality, operationalized as an interaction between its two subcomponents negative affectivity (NA) and social inhibition (SI). We constructed this interaction according to four approaches: (1) sum score product; (2) single product indicator; (3) matched product indicators; and (4) latent moderated structural equations (LMS). In Study 2, we compared these interaction models in a simulation study by assessing for each method the bias and precision of the estimated interaction effect under varying conditions. In Study 1, all methods showed a significant Type D effect on both depression and anxiety, although this effect diminished after including the NA and SI quadratic effects. Study 2 showed that the LMS approach performed best with respect to minimizing bias and maximizing power, even when item scores were ordinal and skewed. However, when latent traits were skewed LMS resulted in more false-positive conclusions, while the Matched PI approach adequately controlled the false-positive rate.

Fitting Large Factor Analysis Models With Ordinal Data

A simulation study was conducted to investigate the model size effect when confirmatory factor analysis (CFA) models include many ordinal items. CFA models including between 15 and 120 ordinal items were analyzed with mean- and variance-adjusted weighted least squares to determine how varying sample size, number of ordered categories, and misspecification affect parameter estimates, standard errors of parameter estimates, and selected fit indices. As the number of items increased, the number of admissible solutions and accuracy of parameter estimates improved, even when models were misspecified. Also, standard errors of parameter estimates were closer to empirical standard deviation values as the number of items increased. When evaluating goodness-of-fit for ordinal CFA with many observed indicators, researchers should be cautious in interpreting the root mean square error of approximation, as this value appeared overly optimistic under misspecified conditions.

Latent variables should remain as such: Evidence from a Monte Carlo study

Use of subject scores as manifest variables to assess the relationship between latent variables produces attenuated estimates. This has been demonstrated for raw scores from classical test theory (CTT) and factor scores derived from factor analysis. Conclusions on scores have not been sufficiently extended to item response theory (IRT) theta estimates, which are still recommended for estimation of relationships between latent variables. This is because IRT estimates appear to have preferable properties compared to CTT, while structural equation modeling (SEM) is often advised as an alternative to scores for estimation of the relationship between latent variables. The present research evaluates the consequences of using subject scores as manifest variables in regression models to test the relationship between latent variables. Raw scores and three methods for obtaining theta estimates were used and compared to latent variable SEM modeling. A Monte Carlo study was designed by manipulating sample size, number of items, type of test, and magnitude of the correlation between latent variables. Results show that, despite the advantage of IRT models in other areas, estimates of the relationship between latent variables are always more accurate when SEM models are used. Recommendations are offered for applied researchers.