Combining Text Mining of Long Constructed Responses and Item-Based Measures: A Hybrid Test Design to Screen for Posttraumatic Stress Disorder (PTSD)

Accurate Assessment via Process Data

Accurate assessment of a student's ability is the key task of a test. Assessments based on final responses are the standard. As the infrastructure advances, substantially more information is observed. One of such instances is the process data that is collected by computer-based interactive items and contain a student's detailed interactive processes. In this paper, we show both theoretically and with simulated and empirical data that appropriately including such information in the assessment will substantially improve relevant assessment precision.

Empirical Priors in Polytomous Computerized Adaptive Tests: Risks and Rewards in Clinical Settings

The use of empirical prior information about participants has been shown to substantially improve the efficiency of computerized adaptive tests (CATs) in educational settings. However, it is unclear how these results translate to clinical settings, where small item banks with highly informative polytomous items often lead to very short CATs. We explored the risks and rewards of using prior information in CAT in two simulation studies, rooted in applied clinical examples. In the first simulation, prior precision and bias in the prior location were manipulated independently. Our results show that a precise personalized prior can meaningfully increase CAT efficiency. However, this reward comes with the potential risk of overconfidence in wrong empirical information (i.e., using a precise severely biased prior), which can lead to unnecessarily long tests, or severely biased estimates. The latter risk can be mitigated by setting a minimum number of items that are to be administered during the CAT, or by setting a less precise prior; be it at the expense of canceling out any efficiency gains. The second simulation, with more realistic bias and precision combinations in the empirical prior, places the prevalence of the potential risks in context. With similar estimation bias, an empirical prior reduced CAT test length, compared to a standard normal prior, in 68% of cases, by a median of 20%; while test length increased in only 3% of cases. The use of prior information in CAT seems to be a feasible and simple method to reduce test burden for patients and clinical practitioners alike.

Text mining to improve screening for trauma-related symptoms in a global sample

Previous studies showed that textual information could be used to screen respondents for posttraumatic stress disorder (PTSD). In this study, we explored the feasibility of using language features extracted from short text descriptions respondents provided of stressful events to predict trauma-related symptoms assessed using the Global Psychotrauma Screen. Texts were analyzed with both closed- and open-vocabulary methods to extract language features representing the occurrence of words, phrases, or specific topics in the description of stressful events. We also evaluated whether combining language features with self-report information, including respondents' demographics, event characteristics, and risk factors for trauma-related disorders, would improve the prediction performance. Data were collected using an online survey on a cross-national sample of 5048 respondents. Results showed that language data achieved the highest predictive power when both closed- and open-vocabulary features were included as predictors. Combining language data and self-report information resulted in a significant increase in performance and in a model which achieved good accuracy as a screener for probable PTSD diagnosis (.7 < AUC ≤ .8), with similar results regardless of the length of the text description of the event. Overall, results indicated that short texts add to the detection of trauma-related symptoms and probable PTSD diagnosis.