The Assessment and Impact of Careless Responding in Routine Outcome Monitoring within Mental Health Care

Detecting non-content-based response styles in survey data: An application of mixture factor analysis

It is common for some participants in self-report surveys to be careless, inattentive, or lacking in effort. Data quality can be severely compromised by responses that are not based on item content (non-content-based [nCB] responses), leading to strong biases in the results of data analysis and misinterpretation of individual scores. In this study, we propose a specification of factor mixture analysis (FMA) to detect nCB responses. We investigated the usefulness and effectiveness of the FMA model in detecting nCB responses using both simulated data (Study 1) and real data (Study 2). In the first study, FMA showed reasonably robust sensitivity (.60 to .86) and excellent specificity (.96 to .99) on mixed-worded scales, suggesting that FMA had superior properties as a screening tool under different sample conditions. However, FMA performance was poor on scales composed of only positive items because of the difficulty in distinguishing acquiescent patterns from valid responses representing high levels of the trait. In Study 2 (real data), FMA detected a minority of cases (6.5%) with highly anomalous response patterns. Removing these cases resulted in a large increase in the fit of the unidimensional model and a substantial reduction in spurious multidimensionality.

Flagging insufficient effort responses in surveys: Stopping rule to prevent insufficient or excessive removal of doubtful data

Detecting participants not exerting sufficient effort (e.g., answering randomly; termed 'insufficient effort responding' or IER) in self-report surveys is crucial but intricate. Challenges of appropriately removing IER resemble a surgeon excising necrotic tissue without being inadequate, excessive, or incorrect. Current cutoff criteria for stopping the elimination of suspected IER responses are often arbitrary. This study proposes an external criterion strategy to identify the optimal cutoff values for various IER detection methods. We investigated the change in correlations between the IER-containing scale (e.g., motivation) and the external criterion (e.g., academic performance, socioeconomic status index, or another questionnaire scale) utilizing simulated and authentic international survey data. The findings revealed that the stopping rule and consequently, the optimal cutoff values for IER detection methods, can be accurately identified by locating the inflection point in the correlation plot with the external criterion. Practical strategies are recommended for applied researchers.

Protocol of a study to benchmark occupational health and safety in Japan: W2S-Ohpm study

We aim to conduct a prospective cohort study to benchmark occupational health and safety in Japan. Here, we describe the detailed protocol for the baseline survey based on the Checklist for Reporting Results of Internet E-Surveys. We conducted the baseline survey for the prospective cohort study in 2022. Our target population was workers in Japan aged 20 years or older, who we sampled to be representative of the Japanese workforce, stratified by sex, age, and region. Among 59,272 registered monitors who answered the initial screening questions, 29,997 completed the survey. After excluding 2,304 invalid responses, we used 27,693 valid participants in our final analysis. The number and mean age of men were 15,201 (55%) and 46 years; those of women were 12,492 (45%) and 45 years. With respect to sex, age, and regional composition, our sampling was representative of Japan's working population. Our sampling for employment status and industry yielded almost the same proportions as a government-led representative sampling of workers in Japan.

Using Mokken scaling techniques to explore carelessness in survey research

Careless responding is a pervasive issue that impacts the interpretation and use of responses from survey instruments. Researchers have proposed numerous useful methods for detecting carelessness in survey research, including relatively simple summary statistics such as the frequency of adjacent responses in the same category (e.g., "long-string" analysis) and outlier statistics (e.g., Mahalanobis distance). Researchers have also used methods based on item response theory (IRT) models to identify examinees whose response patterns are unexpected given item parameters. However, researchers have not fully considered the use of nonparametric IRT methods based on Mokken scale analysis (MSA) to detect carelessness in survey research. MSA is a promising framework in which to consider participant carelessness because it is well suited to contexts in which parametric IRT models may not be appropriate, while still maintaining a focus on fundamental measurement requirements. We used a real data analysis and a simulation study to examine the sensitivity of MSA indicators of response quality to examinee carelessness and compared the results to those from standalone indicators. We also examined the impact of carelessness on the sensitivity of MSA item quality indicators. Numeric and graphical indicators of response quality from MSA indicators were sensitive to examinee carelessness. Graphical displays of nonparametric person response functions (PRFs) provided supplementary insight that can alert researchers to potentially problematic responses. Our results also indicated that MSA indicators of item quality are robust to the presence of participant carelessness. We consider the implications of our findings for research and practice.

Dealing with Careless Responding in Survey Data: Prevention, Identification, and Recommended Best Practices

Surveys administered online have several benefits, but they are particularly prone to careless responding, which occurs when respondents fail to read item content or give sufficient attention, resulting in raw data that may not accurately reflect respondents' true levels of the constructs being measured. Careless responding can lead to various psychometric issues, potentially impacting any area of psychology that uses self-reported surveys and assessments. This review synthesizes the careless responding literature to provide a comprehensive understanding of careless responding and ways to prevent, identify, report, and clean careless responding from data sets. Further, we include recommendations for different levels of screening for careless responses. Finally, we highlight some of the most promising areas for future work on careless responding.

How a Few Inconsistent Respondents Can Confound the Structure of Personality Survey Data

Abstract. In survey data, inconsistent responses due to careless/insufficient effort (C/IE) can lead to problems of replicability and validity. However, data cleaning prior to the main analyses is not yet a standard practice. We investigated the effect of C/IE responses on the structure of personality survey data. For this purpose, we analyzed the structure of the Core-Self Evaluations scale (CSE-S), including the detection of aberrant responses in the study design. While the original theoretical model of the CSE-S assumes that the construct is unidimensional ( Judge et al., 2003 ), recent studies have argued for a multidimensional solution (positive CSE and negative CSE). We hypothesized that this multidimensionality is not substantive but a result of the tendency of C/IE data to generate spurious dimensions. We estimated the confirmatory models before and after removing highly inconsistent response vectors in two independent samples (6% and 4.7%). The analysis of the raw samples clearly favored retaining the two-dimensional model. In contrast, the analysis of the clean datasets suggested the retention of a single factor. A mere 6% C/IE response rate showed enough power to confound the results of the factor analysis. This result suggests that the factor structure of positive and negative CSE factors is spurious, resulting from uncontrolled wording variance produced by a limited proportion of highly inconsistent response vectors. We encourage researchers to include screening for inconsistent responses in their research designs.

Using a consistency check during data collection to identify invalid responding in an online cannabis screening survey

BACKGROUND

Inconsistent responding is a type of invalid responding, which occurs on self-report surveys and threatens the reliability and validity of study results. This secondary analysis evaluated the utility of identifying inconsistent responses as a real-time, direct method to improve quality during data collection for an Internet-based RCT.

METHODS

The cannabis subscale of the Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) was administered as part of eligibility screening for the RCT. Following the consent procedure, the cannabis subscale was repeated during the baseline interview. Responses were automatically compared and individuals with inconsistent responses were screened out.

RESULTS

Nearly half of those initially eligible for the RCT were subsequently screened out for data quality issues (n = 626, 45.3%). Between-group bivariate analysis found that those screened out (OUT) were significantly older (OUT = 39.5 years (SD = 13.9), IN = 35.7 years (SD = 12.9), p < .001), more had annual incomes less than $20,000CND (OUT = 58.3%, IN = 53.0%, p = .047), used cannabis less often in the past 30 days (OUT = 23.3 days (SD = 9.7), IN = 24.8 days (SD = 11.3), p < .006), and had lower total ASSIST scores at screener (OUT = 19.3 (SD = 8.0), IN = 23.8 (SD = 10.4), p < .001) and baseline (OUT = 17.5 (SD = 7.9), IN = 23.3 (SD = 10.3), p < .001) compared to participants who were screened in to the RCT.

CONCLUSION

Inconsistent responding may occur at high rates in Internet research and direct methods to identify invalid responses are needed. Comparing responses for consistency can be programmed in Internet surveys to automatically screen participants during recruitment and reduce the need for post-hoc data cleaning.

Translation, Cultural Adaptation, and Reproducibility of the Physical Activity Readiness Questionnaire for Everyone (PAR-Q+): The Brazilian Portuguese Version

Background: The Physical Activity Readiness Questionnaire for Everyone (PAR-Q+) is the international standard for pre-participation risk stratification and screening. In order to provide a practical and valid screening tool to facilitate safe engagement in physical activity and fitness assessments for the Brazilian population, this study aimed to translate, culturally adapt, and verify the reproducibility of the evidence-based PAR-Q+ to the Brazilian Portuguese language.

Method: Initially, the document was translated by two independent translators, before Brazilian experts in health and physical activity evaluated the translations and produced a common initial version. Next, two English native speakers, fluent in Brazilian Portuguese and accustomed to the local culture, back-translated the questionnaire. These back translations were assessed by the organization in charge of the PAR-Q+, then a final Brazilian version was approved. A total of 493 Brazilians between 5 and 93 yr (39.9 ± 25.4 yr), 59% female, with varying levels of health and physical activity, completed the questionnaire twice, in person or online, 1–2 weeks apart. Cronbach's alpha was used to calculate the internal consistency of all items of the questionnaire, and the Kappa statistic was used to assess the individual reproducibility of each item of the document. Additionally, the intraclass correlation coefficient and its 95% confidence interval (CI) were used to verify the general reproducibility (reliability) of the translated version.

Results: The Brazilian version had an excellent internal consistency (0.993), with an almost perfect agreement in 93.8% of the questions, and a substantial agreement in the other 6.2%. The translated version also had a good to excellent total reproducibility (0.901, 95% CI: 0.887–0.914).

Conclusion: The results show this translation is a valid and reliable screening tool, which may facilitate a larger number of Brazilians to start or increase physical activity participation in a safe manner.

Psychological recovery in a step 4 service: a qualitative study exploring the views of service users and clinicians

BACKGROUND

Psychological recovery is typically measured in improving access to psychological therapies (IAPT) services as the reduction of symptoms on routine outcome measures (ROMs). However, conceptualisations of psychological recovery vary, and there are concerns that ROMs may not be suitable for measuring recovery from complex mental health difficulties.

AIM

To gain insight into psychological recovery and measurement in clinical practice.

METHOD

A qualitative study at a step-4 IAPT service in the North of England. Eight clinicians contributed to focus groups, and 10 service users were later interviewed individually.

RESULTS

Factors that informed four congruent and interacting themes of recovery were identified among participants as an ongoing personal process, of intra-therapeutic and extra-therapeutic factors impacting on recovery in helpful and unhelpful ways, and the need for idiographic long-term recovery measures. The functional and long-term nature of recovery and the socio-economic context for recovery were considered particularly crucial.

CONCLUSIONS

The need for idiographic long-term recovery measures to support the ongoing process of recovery was reported in the majority of instances. Study findings mirrored existing recovery and outcome measurement literature, provided new evidence in relation to recovery from complex mental health difficulties, and made recommendations for future research and clinical practice.

A little garbage in, lots of garbage out: Assessing the impact of careless responding in personality survey data

In self-report surveys, it is common that some individuals do not pay enough attention and effort to give valid responses. Our aim was to investigate the extent to which careless and insufficient effort responding contributes to the biasing of data. We performed analyses of dimensionality, internal structure, and data reliability of four personality scales (extroversion, conscientiousness, stability, and dispositional optimism) in two independent samples. In order to identify careless/insufficient effort (C/IE) respondents, we used a factor mixture model (FMM) designed to detect inconsistencies of response to items with different semantic polarity. The FMM identified between 4.4% and 10% of C/IE cases, depending on the scale and the sample examined. In the complete samples, all the theoretical models obtained an unacceptable fit, forcing the rejection of the starting hypothesis and making additional wording factors necessary. In the clean samples, all the theoretical models fitted satisfactorily, and the wording factors practically disappeared. Trait estimates in the clean samples were between 4.5% and 11.8% more accurate than in the complete samples. These results show that a limited amount of C/IE data can lead to a drastic deterioration in the fit of the theoretical model, produce large amounts of spurious variance, raise serious doubts about the dimensionality and internal structure of the data, and reduce the reliability with which the trait scores of all surveyed are estimated. Identifying and filtering C/IE responses is necessary to ensure the validity of research results.

Determining at What Age Children Provide Sound Self-Reports: An Illustration of the Validity-Index Approach

In psychological assessment of children, it is pivotal to establish from what age on self-reports can complement or replace informant reports. We introduce a psychometric approach to estimate the minimum age for a child to produce self-report data that is of similar quality as informant data. The approach makes use of statistical validity indicators such as person-fit and long-string indices, and can be readily applied to data commonly collected in psychometric studies of child measures. We evaluate and illustrate the approach, using self-report and informant-report data of the PedsQL, a pediatric health-related quality of life measure, from 651 child-mother pairs. To evaluate the approach, we tested various hypotheses about the validity of the self-report data, using the Gnp person-fit index as the validity indicator and the mother informant-data as a benchmark for validity. Results showed that Gnp discriminated between self-reports of younger and older children, between self-reports of children that completed the PedsQL alone or with a parent, and between self-reports and informant reports. We conclude that the validity-index approach has good potential for future applications. Future research should further evaluate the approach for different types of questionnaires (e.g., personality inventories) and using different validity indices (e.g., response-bias indices).