Evidence-Based Assessment of ADHD in Youth Using a Receiver Operating Characteristic Approach

Review of Studies on Incremental Validity of Assessment Measures Used in Psychological Assessment of Attention-Deficit Hyperactivity Disorder

Few studies have summarized the literature relevant to the incremental validity of tools and procedures for the assessment of attention-deficit hyperactivity disorder (ADHD). The current project reviewed such studies published in the prior 18 years. Results from studies on the incremental validity of measures used in the assessment of ADHD were reviewed. Measures included symptom reports, clinical interviews, behavioral observation, continuous performance and other psychomotor tasks, intelligence tests, and measures of executive function. Twenty-nine published studies and two reviews were identified from 2004 to 2022. Incremental validity was determined using various statistics including R2, classification metrics, odds ratios, and post-test probabilities. Findings suggest that symptom reports from a collateral source and continuous performance test measures have incremental validity over self-reports and clinical interviews. Measures of intellectual and executive functioning did not show incremental validity in the diagnosis of ADHD. Findings are relevant to the practitioner, as they bear on the optimization of both the cost-effectiveness and the diagnostic accuracy of combined procedures in the assessment of ADHD.

Systematic Review and Meta-Analysis: Clinical Utility of Continuous Performance Tests for the Identification of Attention-Deficit/Hyperactivity Disorder

OBJECTIVE

We aimed to quantify the clinical utility of continuous performance tests (CPTs) for the diagnosis of attention-deficit/hyperactivity disorder (ADHD) compared to a clinical diagnosis in children and adolescents.

METHOD

Four databases (MEDLINE, PsycINFO, EMBASE, and PubMed) were screened until January 2023. Risk of bias of included results was judged with the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2). We statistically pooled the area under the curve, the sensitivity, and the specificity of 3 commonly used CPTs subscales: omission/inattention, commission/impulsivity, and total number of errors/ADHD subscales (PROSPERO registration: CRD42020168091).

RESULTS

A total of 19 studies using commercially available CPTs were identified. Results from up to 835 control individuals and 819 cases were combined in the summary receiver operating characteristic (ROC) curve analyses (sensitivity and specificity pooling), and up to 996 cases and 1,083 control individuals in the area under the curve (AUC) analyses. Clinical utility as measured by AUCs could be considered as barely acceptable (between 0.7 and 0.8) for the most part, with the best results for the total/ADHD score, followed by omissions/inattention, and poorest for commission/impulsivity scores. A similar pattern was found when pooling sensitivity and specificity: 0.75 (95% CI = 0.66-0.82) and 0.71 (0.62-0.78) for the total/ADHD score; 0.63 (0.49-0.75) and 0.74 (0.65-0.81) for omissions; and 0.59 (0.38-0.77) and 0.66 (CI = 0.50-0.78) for commissions.

CONCLUSION

At the clinical level, CPTs as a stand-alone tool have only a modest to moderate ability to differentiate ADHD from non-ADHD samples. Hence, they should be used only within a more comprehensive diagnostic process.

STUDY PREREGISTRATION INFORMATION

A systematic review of screening tools for ADHD in children and adolescents; https://www.crd.york.ac.uk/prospero/; CRD42020168091.

Is the Conners' continuous performance test helpful for assessing attention deficit hyperactivity disorder in a clinical setting?

BACKGROUND

Despite lacking validation for Norwegian populations, the Conners Continuous Performance Test II (CCPT-II) is applied to almost one-third of children receiving an ADHD diagnosis. However, evidence of the CCPT-II's ability to differentiate between children with and without ADHD is contradictory. Thus, this study examines how CCPT-II results correlate with ADHD symptoms reported by mothers and teachers in a sample representing ordinary child and adolescent mental health services and explores the extent to which the CCPT-II influences the diagnostic result.

METHODS

Correlations between CCPT-II results and ADHD Rating Scale scores and a clinical diagnosis of ADHD were analysed in children aged 6-15 years (N = 69) referred to a child and adolescent psychiatric outpatient clinic.

RESULTS

Total ADHD symptom scores rated by mothers correlated with hit reaction time (HRT) block change (.260), HRT inter-stimulus interval (ISI) change (.264) and CCPT-II overall index (.263), while hyperactivity subscale scores correlated with omissions (.285), HRT (.414) and variability (.400). In teachers' ratings, total ADHD and both subscale scores correlated with commissions (.280-.382), while hyperactivity scores correlated with variability (.265). A higher number of commissions was the only significant difference in CCPT-II performance between children diagnosed with and children without ADHD.

CONCLUSIONS

Correlations between CCPT-II results and ADHD symptoms were all small to moderate. As such, CCPT-II results should be interpreted with caution, because they correspond to a limited degree with other sources of information.

Evidence-Based Assessment for Attention-Deficit/Hyperactivity Disorder

Attention-deficit/hyperactivity disorder (ADHD) is a neurodevelopmental disorder that affects individuals from all life stages, genders, and races/ethnicities. Accurate assessment of ADHD across different populations is essential as undiagnosed ADHD is associated with numerous costly negative public health outcomes and is complicated by high comorbidity and developmental change in symptoms over time. Predictive analysis suggests that best-practice evidence-based assessment of ADHD should include both ADHD-specific and broadband rating scales from multiple informants with consideration of IQ, academic achievement, and executive function when there are concerns about learning. For children under age 12, parent and teacher ratings should be averaged. For adolescents and adults, informant reports should be prioritized when self- and other-report are inconsistent. Future research should provide more stringent evaluation of the sensitivity of measures to treatment response and developmental change over time as well as further validate measures on historically understudied populations (i.e., adults, women, and racial/ethnic minorities).

Using an objective computer task (QbTest) to aid the identification of attention deficit hyperactivity disorder (ADHD) in the Children and Young People Secure Estate (CYPSE): a feasibility randomised controlled trial

OBJECTIVES

QbTest has been shown to improve time to decision/diagnosis for young people with attention deficit hyperactivity disorder (ADHD). The aim was to assess the feasibility of QbTest for young people in prison.

DESIGN

Single-centre feasibility randomised controlled trial (RCT), with 1:1 allocation. Concealed random allocation using an online pseudorandom list with random permuted blocks of varying sizes.

SETTING

One Young Offenders Institution in England.

PARTICIPANTS

355 young people aged 15-18 years displaying possible symptoms of ADHD were assessed for eligibility, 69 were eligible to take part and 60 were randomised.

INTERVENTION

QbTest-a computer task measuring attention, activity and impulsivity.

MAIN OUTCOME MEASURES

Eligibility, recruitment and retention rates and acceptability of randomisation and trial participation.

RESULTS

Of the 355 young people assessed for eligibility, 69 were eligible and 60 were randomised (n=30 QbTest plus usual care; n=30 usual care alone). The study achieved the specified recruitment target. Trial participation and randomisation were deemed acceptable by the majority of participants. 78% of young people were followed up at 3 months, but only 32% at 6 months, although this was also affected by COVID-19 restrictions. Secondary outcomes were mixed. Participants including clinical staff were mostly supportive of the study and QbTest; however, some young people found QbTest hard and there were issues with implementation of the ADHD care pathway. There were no serious adverse events secondary to the study or intervention and no one was withdrawn from the study due to an adverse event.

CONCLUSIONS

With adaptations, a fully powered RCT may be achievable to evaluate the effectiveness of QbTest in the assessment of ADHD in the Children and Young People Secure Estate, with time to decision (days) as the primary outcome measure. However, further programme developmental work is required to address some of the challenges highlighted prior to a larger trial.

TRIAL REGISTRATION NUMBER

ISRCTN17402196.

Ahead of the (ROC) Curve: A Statistical Approach to Utilizing Ex-Gaussian Parameters of Reaction Time in Diagnosing ADHD Across Three Developmental Periods

INTRODUCTION

Performance on executive function (EF) tasks is only modestly predictive of a diagnosis of Attention Deficit Hyperactivity Disorder (ADHD), despite the common assumption that EF deficits are ubiquitous to the disorder. The current study sought to determine whether ex-Gaussian parameters of simple reaction time are better able to discriminate between children and adults with and without ADHD, compared with traditional measures of inhibitory control.

METHODS

Receiver Operating Characteristic (ROC) analyses and the area under the curve (AUC) were used to examine the ability of performance on two commonly used tasks of inhibitory control (i.e. stop signal reaction time (SSRT) and go-no-go tasks) to predict ADHD status in preschool (N = 108), middle childhood (N = 309), and young adulthood (N = 133).

RESULTS

Across all samples, SSRT, go-no-go percentage of failed inhibits, and standard deviation of reaction (SDRT) time to "go" trials, all successfully discriminated between individuals with and without ADHD. Ex-Gaussian decomposition of the RT distribution indicated that both larger tau and larger sigma drove findings for SDRT. Contrary to predictions, traditional measures of inhibitory control were equal if not better predictors of ADHD status than ex-Gaussian parameters.

CONCLUSIONS

Findings support ongoing work to quantify the separate contributions of cognitive subprocesses that drive task performance, which in turn is critical to developing and improving process-based approaches in clinical assessment.

Psychological and neuropsychological underpinnings of attention-deficit/hyperactivity disorder assessment

The identification and diagnosis of attention-deficit/hyperactivity disorder (ADHD) is extremely important in order to help change the trajectory of an individual's life outcomes. A review of the current state of evidence-based assessment of ADHD is dominated by the DSM-5's conceptualization of behaviorally-oriented diagnostic criteria. This assumption that the DSM-5's method for identifying ADHD is the gold standard underlies the research base that evaluates the incremental validity of measures and methods for diagnosing it. That is, when evaluating whether a measure is useful in the identification of ADHD, the 'right answer' is based on the DSM-5's behaviorally-oriented definition. An alternative model for considering the fact that ADHD is a neurodevelopmental disorder, with its roots in executive dysfunction, is proposed. Using neuropsychological and cognitive tests to identify executive functioning problems can be combined with rating scales and interviews to diagnose ADHD in a way that does not ascribe entirely to a behavioral definition of the disorder.

Validity of Visual and Auditory Attention Tests for Detecting ADHD

Objective: This investigation examines the discriminative validity of visual and auditory attention tests for differentiating patients with ADHD from healthy control participants. Method: A total of 107 ADHD patients and 58 healthy control participants were recruited. Visual and auditory attention profiles were obtained using the Conners' Continuous Performance Test 3rd Edition (CPT3) and Conners' Continuous Auditory Test of Attention (CATA), respectively. Results: We found that ADHD patients underperformed healthy controls on all CPT3 and CATA indexes, except Response Style and Hit Reaction Time. The CPT3, CATA, and CPT3 plus CATA all significantly differentiate ADHD patients and controls. CPT3 plus CATA had a greater sensitivity (82.6%), specificity (76%), positive predictive value (88.8%), negative predictive value (65.5%), and overall correct classification rate (80.6%) than CPT3 or CATA alone. Conclusion: Neuropsychological tests CPT3 and CATA provide objective information about cases of ADHD and should be used routinely for clinical assessment.

Caution When Screening for Autism among Socially Anxious Youth

Social anxiety disorder (SAD) is commonly comorbid with autism spectrum disorder (ASD). Here, in a sample of 86 children and adolescents (MAGE = 12.62 years; 68.6% male), 28 of whom were diagnosed with ASD, 34 with SAD, and 24 with comorbid ASD and SAD, we compared parent-reported scores from the Social Responsiveness Scale-Second Edition (SRS-2; Constantino and Gruber in Social Responsiveness Scale (SRS; Constantino and Gruber 2012) to determine the sensitivity and specificity of the measure in cases of differential diagnosis between SAD and ASD. Results suggest that neither the subscales, nor the SRS-2 total score, consistently differed between ASD and SAD. Sensitivity and specificity analyses suggested that the SRS-2 total poorly discriminated ASD from SAD. When screening socially anxious youth for possible ASD, caution should be taken.

DAS-II Cognitive Profiles Are Not Diagnostically Meaningful For Autism: A ROC Analysis

Intelligence assessment is an integral part of a comprehensive autism evaluation. Many past studies have described a cognitive profile of autistic individuals characterized by higher nonverbal than verbal IQ scores. The diagnostic utility of this profile, however, remains unknown. We leveraged receiver operating characteristic methods to determine the sensitivity, specificity, and area under the curve (AUC) of three different IQ profiles in a large sample of children who have an autism spectrum disorder diagnosis (N = 1,228, Simons Simplex Collection) who completed the Differential Ability Scales-Second Edition (DAS-II), School Age compared to the normative sample provided by the DAS-II publisher (N = 2,200). The frequently discussed nonverbal > verbal IQ profile performed near chance at distinguishing ASD from normative individuals (AUC: 0.54, 95% CI [0.52-0.56]), and performed significantly worse for females than males (AUC: females: 0.46 [0.41-0.52]; males: 0.55 [0.53-0.58]). All cognitive profiles showed AUC < 0.56. We conclude that while significant differences between verbal and nonverbal IQ scores exist at the group level, these differences are small in an absolute sense and not meaningful at an individual level. We do not recommend using cognitive profiles to aid in autism diagnostic decision-making. LAY SUMMARY: Some researchers and clinicians have reported an "autistic cognitive profile" of higher nonverbal intelligence than verbal intelligence. In an analysis of over 1,000 autistic children, we found that the group's average nonverbal intelligence is usually higher than their verbal intelligence. However, this pattern should not be used by clinicians to make an individual diagnosis of autism because our results show it is not helpful nor accurate.

Sluggish Cognitive Tempo in Autism Spectrum Disorder, ADHD, and Their Comorbidity: Implications for Impairment

Objective: Sluggish Cognitive Tempo (SCT), characterized by lethargy and daydreaming, has most commonly been studied in community samples and in youth with Attention-Deficit/Hyperactivity Disorder (ADHD). Despite shared neurodevelopmental symptoms with ADHD, few studies have investigated SCT in Autism Spectrum Disorders (ASD). The current study investigated SCT symptoms in youth with ASD, ADHD, and comorbid ASD+ADHD to explore the relations between SCT and global and social impairment.Method: Caregivers of children and adolescents (n = 98; ages 6-17) diagnosed with ADHD (n = 46), ASD (n = 28), or ASD+ADHD (n = 24) completed measures of social impairment, SCT, and demographic variables.Results: All three clinical groups demonstrated comparable levels of SCT. Diagnosis and SCT independently contributed to parent-rated social impairment, while SCT and IQ, but not diagnosis, contributed to clinician-rated global functioning. Specifically, having comorbid ASD+ADHD, but not an ASD or ADHD diagnosis alone, significantly predicted greater social impairment.Conclusion: These results extend previous literature investigating SCT in ASD and provide evidence to suggest that SCT is associated with social and global impairment above and beyond the impairment associated with ADHD and/or ASD. These results may have implications for clinical assessment and treatment of ASD and ADHD.

Translating Cognitive Vulnerability Theory Into Improved Adolescent Depression Screening: A Receiver Operating Characteristic Approach

Traditionally, screening research tests how well a given symptom inventory can identify a concurrent depressive episode. Although developmental psychopathology could inform screening protocols for a myriad of depression outcomes (e.g., prospective depressive episodes), approaches typically used in research make it difficult to translate these findings. Using a translational analytic approach and multiwave longitudinal study design, we examined how screening for cognitive vulnerabilities (rumination, dysfunctional attitudes, and attributional style) may improve our ability to identify concurrent depressive episodes, prospective depressive episodes, first lifetime episodes of depression, and recurrent major depressive episodes. There were 473 sixth-grade (early adolescents) and ninth-grade (middle adolescents; AgeM = 13.15, AgeSD = 1.62) students who completed baseline self-report cognitive vulnerability and depressive symptom measures. At baseline and every 6 months for 3 years, pediatric depression interviews were completed by the caregiver and youth. A receiver operating characteristic (ROC) approach was utilized to test our aims. Distinct algorithms best forecasted our depression outcomes. Rumination and attributional style emerged as unique and incrementally valid predictors for prospective episodes after controlling for baseline depressive symptoms. Rumination was the only unique predictor for first lifetime depressive episodes. For recurrent major depression, rumination in early adolescence and attributional style in middle adolescence served as incremental predictors beyond baseline depressive symptoms. Proposed cutoffs and diagnostic likelihood ratios are offered for algorithms for each depression outcome. Assessing cognitive vulnerability represents a feasible method to improve depression screening initiatives. Using an ROC-informed approach can help prevention initiatives better leverage the considerable gains made within developmental psychopathology research.

Informant discrepancies in assessments of psychosocial functioning in school-based services and research: Review and directions for future research

Psychosocial functioning plays a key role in students' wellbeing and performance inside and outside of school. As such, techniques designed to measure and improve psychosocial functioning factor prominently in school-based service delivery and research. Given that the different contexts (e.g., school, home, community) in which students exist vary in the degree to which they influence psychosocial functioning, educators and researchers often rely on multiple informants to characterize intervention targets, monitor intervention progress, and inform the selection of evidence-based services. These informants include teachers, students, and parents. Across research teams, domains, and measurement methodologies, researchers commonly observe discrepancies among informants' reports. We review theory and research-occurring largely outside of school-based service delivery and research-that demonstrates how patterns of informant discrepancies represent meaningful differences that can inform our understanding of psychosocial functioning. In turn, we advance a research agenda to improve use and interpretation of informant discrepancies in school-based services and research.

The Work and Social Adjustment Scale for Youth: A Measure for Assessing Youth Psychosocial Impairment Regardless of Mental Health Status

A key component of delivering mental health services involves evaluating psychosocial impairments linked to mental health concerns. Youth may experience these impairments in various ways (e.g., dysfunctional family and/or peer relationships, poor school performance). Importantly, youth may display symptoms of mental illness without co-occurring psychosocial impairments, and the reverse may be true. However, all available instruments for assessing youth psychosocial impairments presume the presence of mental health concerns among those assessed. Consequently, key gaps exist in knowledge about the developmental psychopathology of psychosocial impairments; and thus how to understand impairments in the context of youth mental health. To address these issues we developed a modified version of a 5-item measure of adult psychosocial impairments (i.e., Work and Social Adjustment Scale for Youth [WSASY]) and tested its psychometric properties. A mixed clinical/community sample of adolescents and parents completed parallel versions of the WSASY, along with a multi-domain, multi-method battery of measures of adolescent internalizing and externalizing concerns, parent psychosocial functioning, adolescent-parent conflict, adolescent peer functioning, and observed social skills. On both versions of the WSASY, increased scores related to increased adolescent mental health concerns, adolescent-parent conflict, parent psychosocial dysfunction, and peer-related impairments. WSASY scores also distinguished adolescents who displayed co-occurring mental health concerns from those who did not, and related to observed social skills deficits within social interactions with unfamiliar peers. The WSASY opens doors to new areas of inquiry regarding the developmental psychopathology of impairment, including questions regarding the onset of impairments and their links to mental health.

Incremental clinical utility of continuous performance tests in childhood ADHD - an evidence-based assessment approach

Despite extensive research on attention deficit hyperactivity disorder (ADHD), there are still uncertainties regarding the clinical utility of different ADHD assessment methods. This study aimed to examine the incremental clinical utility of Conners’ continuous performance test (CPT) II and QbTest in diagnostic assessments and treatment monitoring of attention deficit hyperactivity disorder (ADHD). Retrospective data from child and adolescent psychiatric records of two populations were studied. The diagnostic clinical utility of Conners’ CPT II and QbTest was analysed using receiver operator characteristics (ROC) and post‐test probability in 80 children with and 38 without ADHD. Dose titrations of central stimulants in 56 children with ADHD were evaluated using QbTest and the Swanson, Nolan, Pelham, version IV (SNAP‐IV) scale. Conners’ CPT II, but not QbTest, had incremental clinical utility in diagnostic assessment of children with ADHD when teacher and parent ratings were inconclusive. QbTest proved useful in titration of central stimulant treatment when parent ratings were inconclusive. Continuous performance tests were found to be clinically useful when rating scales were inconclusive.

Accuracy of Achenbach Scales in the Screening of Attention-Deficit/Hyperactivity Disorder in a Community Mental Health Clinic

OBJECTIVE

To use receiver-operating characteristics analysis to identify multilevel diagnostic likelihood ratios and provide a framework for the diagnosis of attention-deficit/hyperactivity disorder (ADHD) in children (5-10 years old) and adolescents (11-18 years old) in an outpatient setting.

METHOD

Caregiver, teacher, and youth reports from the Achenbach System of Empirically Based Assessment (ASEBA) were obtained for 299 children and 321 adolescents with multiple imputation of missing data. The reference standard was diagnosis of ADHD based on case history and a semistructured diagnostic interview masked to the ASEBA measurements.

RESULTS

In children, caregiver-reported Attention Problems (area under the curve [AUC] = 0.74) outperformed all other subscales of the caregiver and teacher measures (AUCs ≤ 0.72). In the older sample, caregiver- and teacher-reported Attention Problems (caregiver AUC = 0.73; teacher AUC = 0.61) were best at identifying ADHD. Inclusion of caregiver and teacher reports significantly (p < .001 for all comparisons) increased prediction of ADHD diagnosis, whereas youth self-report did not.

CONCLUSION

Caregiver-reported Attention Problems were more useful than teacher-reports and self-report in identifying ADHD. Combining caregiver and teacher reports improved identification. Multilevel likelihood ratios are provided to facilitate routine clinical use.