The Work and Social Adjustment Scale for Youth: Psychometric properties of the teacher version and evidence of contextual variability in psychosocial impairments

The Operations Triad Model and Youth Mental Health Assessments: Catalyzing a Paradigm Shift in Measurement Validation

Researchers strategically assess youth mental health by soliciting reports from multiple informants. Typically, these informants (e.g., parents, teachers, youth themselves) vary in the social contexts where they observe youth. Decades of research reveal that the most common data conditions produced with this approach consist of discrepancies across informants' reports (i.e., informant discrepancies). Researchers should arguably treat these informant discrepancies as domain-relevant information: data relevant to understanding youth mental health domains (e.g., anxiety, depression, aggression). Yet, historically, in youth mental health research as in many other research areas, one set of paradigms has guided interpretations of informant discrepancies: Converging Operations and the Multi-Trait Multi-Method Matrix (MTMM). These paradigms (a) emphasize shared or common variance observed in multivariate data, and (b) inspire research practices that treat unique variance (i.e., informant discrepancies) as measurement confounds, namely random error and/or rater biases. Several yearsw ago, the Operations Triad Model emerged to address a conceptual problem that Converging Operations does not address: Some informant discrepancies might reflect measurement confounds, whereas others reflect domain-relevant information. However, addressing this problem requires more than a conceptual paradigm shift beyond Converging Operations. This problem necessitates a paradigm shift in measurement validation. We advance a paradigm (Classifying Observations Necessitates Theory, Epistemology, and Testing [CONTEXT]) that addresses problems with using the MTMM in youth mental health research. CONTEXT optimizes measurement validity by guiding researchers to leverage (a) informants that produce domain-relevant informant discrepancies, (b) analytic procedures that retain domain-relevant informant discrepancies, and (c) study designs that facilitate detecting domain-relevant informant discrepancies.

Multi-Informant Ratings of Childhood Limited Prosocial Emotions: Mother, Father, and Teacher Perspectives

OBJECTIVE

Modest agreement between mothers', fathers', and teachers' reports of child psychopathology can cause diagnostic ambiguity. Despite this, there is little research on informant perspectives of youth's limited prosocial emotions (LPEs). We examined the relationship between mother-, father-, and teacher-reported LPE in a clinical sample of elementary school-aged children.

METHOD

The sample included 207 primarily Caucasian (n = 175, 84.5%) children (136 boys; 65.7%) aged 6-13 years (M = 8.35, SD = 2.04) referred to an outpatient child diagnostic clinic focused on externalizing problems. We report the percentage of youth meeting LPE criteria as a function of informant perspective(s). Utilizing standard scores, we report distributions of informant dyads in agreement/disagreement regarding child LPE, followed up by polynomial regressions to further interrogate the relationship between mother-, father-, and teacher-reported LPE as it relates to conduct problems (CPs).

RESULTS

The prevalence of child LPE was approximately twice as large when compared to those reported in community samples; mothers and fathers generally agreed on their child's LPE symptoms (55% agreement). Higher-order nonlinear interactions between mothers and fathers, as well as parents and teachers, emerged; discrepancies between informants, characterized by low levels of LPE reported by the child's mother, were predictive of youth at the highest risk for CPs.

CONCLUSIONS

Our findings emphasize the clinical utility of gathering multiple reports of LPE when serious CPs are suspected. It may be beneficial for clinicians to give significant consideration to teacher reported LPE when interpreting multiple-informant reports of LPE.

Introduction to the Special Issue. A Dozen Years of Demonstrating That Informant Discrepancies are More Than Measurement Error: Toward Guidelines for Integrating Data from Multi-Informant Assessments of Youth Mental Health

Validly characterizing youth mental health phenomena requires evidence-based approaches to assessment. An evidence-based assessment cannot rely on a "gold standard" instrument but rather, batteries of instruments. These batteries include multiple modalities of instrumentation (e.g., surveys, interviews, performance-based tasks, physiological readings, structured clinical observations). Among these instruments are those that require soliciting reports from multiple informants: People who provide psychometrically sound data about youth mental health (e.g., parents, teachers, youth themselves). The January 2011 issue of the Journal of Clinical Child and Adolescent Psychology (JCCAP) included a Special Section devoted to the most common outcome of multi-informant assessments of youth mental health, namely discrepancies across informants' reports (i.e., informant discrepancies). The 2011 JCCAP Special Section revolved around a critical question: Might informant discrepancies contain data relevant to understanding youth mental health (i.e., domain-relevant information)? This Special Issue is a "sequel" to the 2011 Special Section. Since 2011, an accumulating body of work indicates that informant discrepancies often contain domain-relevant information. Ultimately, we designed this Special Issue to lay the conceptual, methodological, and empirical foundations of guidelines for integrating multi-informant data when informant discrepancies contain domain-relevant information. In this introduction to the Special Issue, we briefly review the last 12 years of research and theory on informant discrepancies. This review highlights limitations inherent to the most commonly used strategies for integrating multi-informant data in youth mental health. We also describe contributions to the Special Issue, including articles about informant discrepancies that traverse multiple content areas (e.g., autism, implementation science, measurement validation, suicide).

Assessing Peer-Related Impairments Linked to Adolescent Social Anxiety: Strategic Selection of Informants Optimizes Prediction of Clinically Relevant Domains

Socially anxious adolescents commonly experience impaired interpersonal functioning with unfamiliar, same-age peers. Yet, we lack short screening tools for assessing peer-related impairments. Recent work revealed that a parent-reported, three-item screening tool produced scores that uniquely related to social anxiety concerns. However, this tool ought to go beyond linking impairments to service needs (i.e., social anxiety symptoms). This tool should also inform the goals of services, in particular by linking impairments to key domains relevant to therapeutically addressing adolescents' anxiety-related needs, such as social skills when interacting with unfamiliar peers. This requires an assessment approach that involves strategic selection of informants who vary in their expertise for observing anxiety-related needs, as well as the therapeutic goals for addressing anxiety-related impairments (e.g., social skills within peer interactions). We leveraged parents' reports to link impairments to social anxiety-related needs. To link impairments to social skills, we leveraged informants (i.e., unfamiliar untrained observers [UUOs]) who observed adolescents within tasks designed to simulate interactions with same-age, unfamiliar peers. We tested this approach using a mixed-clinical/community sample of 134 adolescents, ages 14- to 15 years old. We leveraged multi-informant survey reports to assess adolescent social anxiety, and trained independent observers rated adolescents' social skills within unfamiliar peer interactions. Parents' reports performed best when distinguishing adolescents on referral status and predicting survey-reported social anxiety, whereas only UUOs' reports predicted independent observers' social skills ratings. These findings inform the strategic selection of informants for assessing impairments that commonly prompt the need for adolescents to access mental health services for social anxiety.

When Adolescents Experience Co-Occurring Social Anxiety and ADHD Symptoms: Links With Social Skills When Interacting With Unfamiliar Peer Confederates

Adolescents with elevated social anxiety commonly experience peer-related impairments - particularly with same-age, unfamiliar peers - stemming from their avoidant behaviors. Yet, peer-related impairments are not unique to social anxiety. For example, adolescents who experience social anxiety may also experience symptoms of attention deficit/hyperactivity disorder (ADHD), which also increase risk for peer-related impairments. Relative to social anxiety, peer-related impairments linked to ADHD symptoms more likely stem from hyperactivity (i.e., approach behaviors). These distinct pathways point to adolescents with elevated social anxiety and ADHD symptoms (i.e., social anxiety + ADHD) experiencing particularly high peer-related impairments, which commonly manifest as behavioral displays of low social skills when interacting with unfamiliar peers. We tested this notion in a mixed-clinical/community sample of 134 14- to 15-year-old adolescents and their parents. Adolescents participated in a series of social interaction tasks designed to simulate how adolescents interact with same-age, unfamiliar peers. Trained observers independently rated adolescents on observed social skills within these interactions. Both parents and adolescents completed parallel surveys of social anxiety and ADHD symptoms, which we used to identify social anxiety + ADHD adolescents as well as other combinations of social anxiety and ADHD symptoms (i.e., neither, elevated on one but not the other). Adolescents with social anxiety + ADHD displayed significantly lower social skills, relative to all other groups. Among adolescents, social anxiety + ADHD may have a compounding effect on social skills. As such, therapists working with social anxiety + ADHD adolescents should probe for peer-related impairments and factors implicated in the development and maintenance of these impairments.

Conceptual, methodological, and measurement factors that disqualify use of measurement invariance techniques to detect informant discrepancies in youth mental health assessments

On page 1 of his classic text, Millsap (2011) states, “Measurement invariance is built on the notion that a measuring device should function the same way across varied conditions, so long as those varied conditions are irrelevant [emphasis added] to the attribute being measured.” By construction, measurement invariance techniques require not only detecting varied conditions but also ruling out that these conditions inform our understanding of measured domains (i.e., conditions that do not contain domain-relevant information). In fact, measurement invariance techniques possess great utility when theory and research inform their application to specific, varied conditions (e.g., cultural, ethnic, or racial background of test respondents) that, if not detected, introduce measurement biases, and, thus, depress measurement validity (e.g., academic achievement and intelligence). Yet, we see emerging bodies of work where scholars have “put the cart before the horse” when it comes to measurement invariance, and they apply these techniques to varied conditions that, in fact, may reflect domain-relevant information. These bodies of work highlight a larger problem in measurement that likely cuts across many areas of scholarship. In one such area, youth mental health, researchers commonly encounter a set of conditions that nullify the use of measurement invariance, namely discrepancies between survey reports completed by multiple informants, such as parents, teachers, and youth themselves (i.e., informant discrepancies). In this paper, we provide an overview of conceptual, methodological, and measurement factors that should prevent researchers from applying measurement invariance techniques to detect informant discrepancies. Along the way, we cite evidence from the last 15 years indicating that informant discrepancies reflect domain-relevant information. We also apply this evidence to recent uses of measurement invariance techniques in youth mental health. Based on prior evidence, we highlight the implications of applying these techniques to multi-informant data, when the informant discrepancies observed within these data might reflect domain-relevant information. We close by calling for a moratorium on applying measurement invariance techniques to detect informant discrepancies in youth mental health assessments. In doing so, we describe how the state of the science would need to fundamentally “flip” to justify applying these techniques to detect informant discrepancies in this area of work.

Integrating multi-informant reports of youth mental health: A construct validation test of Kraemer and colleagues’ (2003) Satellite Model

Accurately assessing youth mental health involves obtaining reports from multiple informants who typically display low levels of correspondence. This low correspondence may reflect situational specificity. That is, youth vary as to where they display mental health concerns and informants vary as to where and from what perspective they observe youth. Despite the frequent need to understand and interpret these informant discrepancies, no consensus guidelines exist for integrating informants’ reports. The path to building these guidelines starts with identifying factors that reliably predict the level and form of these informant discrepancies, and do so for theoretically and empirically relevant reasons. Yet, despite the knowledge of situational specificity, few approaches to integrating multi-informant data are well-equipped to account for these factors in measurement, and those that claim to be well-positioned to do so have undergone little empirical scrutiny. One promising approach was developed roughly 20 years ago by Kraemer and colleagues (2003). Their Satellite Model leverages principal components analysis (PCA) and strategic selection of informants to instantiate situational specificity in measurement, namely components reflecting variance attributable to the context in which informants observe behavior (e.g., home/non-home), the perspective from which they observe behavior (e.g., self/other), and behavior that manifests across contexts and perspectives (i.e., trait). The current study represents the first construct validation test of the Satellite Model. A mixed-clinical/community sample of 134 adolescents and their parents completed six parallel surveys of adolescent mental health. Adolescents also participated in a series of simulated social interactions with research personnel trained to act as same-age, unfamiliar peers. A third informant (unfamiliar untrained observer) viewed these interactions and completed the same surveys as parents and adolescents. We applied the Satellite Model to each set of surveys and observed high internal consistency estimates for each of the six-item trait (α = 0.90), context (α = 0.84), and perspective (α = 0.83) components. Scores reflecting the trait, context, and perspective components displayed distinct patterns of relations to a battery of criterion variables that varied in the context, perspective, and source of measurement. The Satellite Model instantiates situational specificity in measurement and facilitates unifying conceptual and measurement models of youth mental health.