Evaluation of the Technical Adequacy of Three Methods for Identifying Specific Learning Disabilities Based on Cognitive Discrepancies

The effect of measurement error on the positive predictive value of PSW methods for SLD identification: How buffer zones dispel the illusion of inaccuracy

Patterns of Strengths and Weaknesses (PSW) methods are widely used for identifying specific learning disabilities (SLDs). Several researchers, however, have reported that the diagnostic accuracy of PSW methods is unacceptably low when strict thresholds were used to identify students with SLDs. We believe these findings give a misleading impression of the magnitude of the diagnostic errors that are likely to arise in PSW assessments. In a simulation study of 10 million cases using a simplified PSW method for demonstration, most of what have been called diagnostic errors were cases in which observed scores and true scores fell on opposite sides of a strict threshold but were still within a buffer zone the size of a typical measurement error. Because small score differences do not result in meaningfully different case conceptualizations, the use of buffer zones reveals that previous estimates of the diagnostic accuracy of PSW methods are misleadingly low. We also demonstrate that diagnostic decisions become increasingly reliable when observed scores are comfortably distant from diagnostic thresholds. For practitioners, we present a flowchart and practical guidelines to improve the accuracy and stability of SLD identification decisions.

Assessment of Specific Learning Disabilities and Intellectual Disabilities

We review literature related to the assessment and identification of Specific Learning Disabilities (SLD) and Intellectual Disabilities (ID). SLD and ID are the only two disorders requiring psychometric test performance for identification within the group of neurodevelopmental disorders in Diagnostic and Statistical Manual - 5. SLD and ID are considered exclusionary of one another, but the processes for assessment and identification of each disorder vary. There is controversy about the identification and assessment methods for SLD, with little consensus. Unlike ID, SLD is weakly related to full-scale IQ, and there is insufficient evidence that the routine assessment of IQ or cognitive skills adds value to SLD identification and treatment. We have proposed a hybrid method based on the assessment of low achievement with norm-referenced tests, instructional response, and other disorders and contextual factors that may be comorbid or contraindicative of SLD. In contrast to SLD, there is strong consensus for a three-prong definition for the identification and assessment of ID: (a) significantly subaverage IQ, (b) adaptive behavior deficits that interfere with independent living in the community, and (c) age of onset in the developmental period. For both SLD and ID, we identify areas of controversy and best practices for identification and assessment.

Remote Assessment: Origins, Benefits, and Concerns

Although guidelines surrounding COVID-19 have relaxed and school-aged students are no longer required to wear masks and social distance in schools, we have become, as a nation and as a society, more comfortable working from home, learning online, and using technology as a platform to communicate ubiquitously across ecological environments. In the school psychology community, we have also become more familiar with assessing students virtually, but at what cost? While there is research suggesting score equivalency between virtual and in-person assessment, score equivalency alone is not sufficient to validate a measure or an adaptation thereof. Furthermore, the majority of psychological measures on the market are normed for in-person administration. In this paper, we will not only review the pitfalls of reliability and validity but will also unpack the ethics of remote assessment as an equitable practice.

Prediction With Mixed Effects Models: A Monte Carlo Simulation Study

Oftentimes in many fields of the social and natural sciences, data are obtained within a nested structure (e.g., students within schools). To effectively analyze data with such a structure, multilevel models are frequently employed. The present study utilizes a Monte Carlo simulation to compare several novel multilevel classification algorithms across several varied data conditions for the purpose of prediction. Among these models, the panel neural network and Bayesian generalized mixed effects model (multilevel Bayes) consistently yielded the highest prediction accuracy in test data across nearly all data conditions.

Cognitive Dimensions of Learning in Children With Problems in Attention, Learning, and Memory

A data-driven, transdiagnostic approach was used to identify the cognitive dimensions linked with learning in a mixed group of 805 children aged 5 to 18 years recognised as having problems in attention, learning and memory by a health or education practitioner. Assessments included phonological processing, information processing speed, short-term and working memory, and executive functions, and attainments in word reading, spelling, and maths. Data reduction methods identified three dimensions of phonological processing, processing speed and executive function for the sample as a whole. This model was comparable for children with and without ADHD. The severity of learning difficulties in literacy was linked with phonological processing skills, and in maths with executive control. Associations between cognition and learning were similar across younger and older children and individuals with and without ADHD, although stronger links between learning-related problems and both executive skills and processing speed were observed in children with ADHD. The results establish clear domain-specific cognitive pathways to learning that distinguish individuals in the heterogeneous population of children struggling to learn.

The influence of language knowledge and test components on reading comprehension scores

This study investigated the dependability of reading comprehension scores across different text genres and response formats for readers with varied language knowledge. Participants included 78 fourth-graders in an urban elementary school. A randomized and counterbalanced 3 × 2 study design investigated three response formats (open-ended, multiple-choice, retell) and two text genres (narrative, expository) from the Qualitative Reading Inventory (QRI-5) reading comprehension test. Standardized language knowledge measures from the Woodcock Johnson III Tests of Achievement (Academic Knowledge, Oral Comprehension, Picture Vocabulary) defined three reader profiles: (a) < 90 as emerging, (b) 90-100 as basic, and (c) > 100 as proficient. Generalizability studies partitioned variance in scores for reader, text genre, and response format for all three groups. Response format accounted for 42.8 to 62.4% of variance in reading comprehension scores across groups, whereas text genre accounted for very little variance (1.2-4.1%). Single scores were well below a 0.80 dependability threshold (absolute phi coefficients = 0.06-0.14). Decision studies projecting dependability achieved with additional scores varied by response format for each language knowledge group, with very low projected dependability on open-ended and multiple-choice scores for readers with basic language knowledge. Multiple-choice scores had similarly low projected dependability levels for readers with emerging language knowledge. Findings evidence interactions between reader language knowledge and response format in reading comprehension assessment practices. Implications underscore the limitations of using a single score to classify readers with and without proficiency in foundational skills.

Muddled Measurement: A Historical Perspective on Questionable Practices in School Psychology’s Assessment of Learning Disabilities

As school psychologists we are well trained with using and interpreting a variety of psychometric instruments, yet there are several studies that indicate that school psychologist’s interpretive practices veer off the intended path. The application of assessment to the process of identification of specific learning disabilities (SLD) has been noted as one of the more problematic areas in psychoeducational assessment for myriad reasons. While the medical field has begun to focus on de-implementing ineffective practices, the field of psychology has not followed as readily. This article considers the costs of poor decision making in the context of SLD evaluation and seeks to identify evidence-based assessment practices for SLD identification and decision making. After considering historical perspectives, approaches and practices for assessing SLD, actuarial interpretation, and treatment validity will be discussed.

Comparison of Reading Growth Among Students With Severe Reading Deficits Who Received Intervention to Typically Achieving Students and Students Receiving Special Education

This study compared the reading growth of students with and without learning disabilities, and students with and without reading deficits in response to tier 2 reading interventions within a response-to-intervention framework. Participants were 499 second- and third-grade students in six urban schools. Students who scored at or below the 10th percentile on the fall reading screening assessment were identified as having a severe reading deficit and received a tier 2 reading intervention that was targeted to their needs. Results showed a significant effect between groups on reading growth. Students with severe reading deficits receiving targeted tier 2 intervention grew at a rate that equaled the rate of growth of students without reading deficits and was significantly higher than students who were receiving special education services for reading. Implications for practice, suggestions for future research, and study limitations are discussed.

The Multiple Deficit Model: Progress, Problems, and Prospects

The multiple deficit model (MDM) was proposed because the prevailing single-deficit model provided an inadequate account of atypical neuropsychological development. Across methods and levels of analysis, there has been support for the two fundamental tenets of the MDM, that multiple predictors contribute probabilistically to neurodevelopmental disorders and shared risk factors contribute to comorbidity. Diagnostically, the multiplicity of factors means that no single cognitive deficit or combination of deficits can be used to rule in or out most neurodevelopmental disorders. Challenges for the MDM are that the theory is difficult to falsify and that current cross-sectional studies cannot establish causality. Prospects for further development of the MDM include incorporating an explicit focus on promotive and protective factors and pursuing mechanistic connections between multiple factors across levels of analysis.

A Misuse of IQ Scores: Using the Dual Discrepancy/Consistency Model for Identifying Specific Learning Disabilities

The purpose of this article is to describe the origins of patterns of strengths and weaknesses (PSW) methods for identifying specific learning disabilities (SLD) and to provide a comprehensive review of the assumptions and evidence supporting the most commonly-used PSW method in the United States: Dual Discrepancy/Consistency (DD/C). Given their use in determining whether students have access to special education and related services, it is important that any method used to identify SLD have supporting evidence. A review of the DD/C evidence indicates it cannot currently be classified as an evidence-based method for identifying individuals with a SLD. We show that the DD/C method is unsound for three major reasons: (a) it requires test scores have properties that they fundamentally lack, (b) lack of experimental utility evidence supporting its use, and (c) evidence supporting the inability of the method to identify SLD accurately.

Inhibition, Updating Working Memory, and Shifting Predict Reading Disability Symptoms in a Hybrid Model: Project KIDS

Recent achievement research suggests that executive function (EF), a set of regulatory processes that control both thought and action necessary for goal-directed behavior, is related to typical and atypical reading performance. This project examines the relation of EF, as measured by its components, Inhibition, Updating Working Memory, and Shifting, with a hybrid model of reading disability (RD). Our sample included 420 children who participated in a broader intervention project when they were in KG-third grade (age M = 6.63 years, SD = 1.04 years, range = 4.79-10.40 years). At the time their EF was assessed, using a parent-report Behavior Rating Inventory of Executive Function (BRIEF), they had a mean age of 13.21 years (SD = 1.54 years; range = 10.47-16.63 years). The hybrid model of RD was operationalized as a composite consisting of four symptoms, and set so that any child could have any one, any two, any three, any four, or none of the symptoms included in the hybrid model. The four symptoms include low word reading achievement, unexpected low word reading achievement, poorer reading comprehension compared to listening comprehension, and dual-discrepancy response-to-intervention, requiring both low achievement and low growth in word reading. The results of our multilevel ordinal logistic regression analyses showed a significant relation between all three components of EF (Inhibition, Updating Working Memory, and Shifting) and the hybrid model of RD, and that the strength of EF's predictive power for RD classification was the highest when RD was modeled as having at least one or more symptoms. Importantly, the chances of being classified as having RD increased as EF performance worsened and decreased as EF performance improved. The question of whether any one EF component would emerge as a superior predictor was also examined and results showed that Inhibition, Updating Working Memory, and Shifting were equally valuable as predictors of the hybrid model of RD. In total, all EF components were significant and equally effective predictors of RD when RD was operationalized using the hybrid model.

Simulating Data for Clinical Research: A Tutorial

Simulation studies use computer-generated data to examine questions of interest that have traditionally been used to study properties of statistics and estimating algorithms. With the recent advent of powerful processing capabilities in affordable computers along with readily usable software, it is now feasible to use a simulation study to aid in clinical decision making. By simulating large quantities of data that mimic clinical situations, it is possible to understand the ramifications of different decisions better than using tangentially relevant data or intuition. In this tutorial article, I describe the general steps in conducting a simulation study with particular emphasis on clinical decision making. I conclude with a didactic example taken from clinical literature on identifying a specific learning disability.

Simulation of LD Identification Accuracy Using a Pattern of Processing Strengths and Weaknesses Method With Multiple Measures

We investigated the classification accuracy of learning disability (LD) identification methods premised on the identification of an intraindividual pattern of processing strengths and weaknesses (PSW) method using multiple indicators for all latent constructs. Known LD status was derived from latent scores; values at the observed level identified LD status for individual cases according to the concordance/discordance method. Agreement with latent status was evaluated using (a) a single indicator, (b) two indicators as part of a test-retest "confirmation" model, and (c) a mean score. Specificity and negative predictive value (NPV) were generally high for single indicators (median specificity = 98.8%, range = 93.4%-99.7%; median NPV = 94.2%, range = 85.6%-98.7%), but low for sensitivity (median sensitivity = 49.1%, range = 20.3%-77.1%) and positive predictive value (PPV; median PPV = 48.8%, range = 23.5%-69.6%). A test-retest procedure produced inconsistent and small improvements in classification accuracy, primarily in "not LD" decisions. Use of a mean score produced small improvements in classifications (mean improvement = 2.0%, range = 0.3%-2.8%). The modest gains in agreement do not justify the additional testing burdens associated with incorporating multiple tests of all constructs.

Let's Not Do Away with Comprehensive Cognitive Assessments Just Yet

We review rational and empirical reasons that comprehensive cognitive assessments are useful sources of information in the evaluation and treatment of learning disabilities. However, the existing evidence base that demonstrates the value of comprehensive cognitive assessments for this purpose is not nearly as strong as it needs to be. Proponents of comprehensive cognitive assessments for learning disability identification must do more to rigorously evaluate their beliefs or else concede the argument to those with better evidence.

Comprehensive Cognitive Assessments are not Necessary for the Identification and Treatment of Learning Disabilities

There is considerable controversy about the necessity of cognitive assessment as part of an evaluation for learning and attention problems. The controversy should be adjudicated through an evaluation of empirical research. We review five sources of evidence commonly provided as support for cognitive assessment as part of the learning disability (LD) identification process, highlighting significant gaps in empirical research and where existing evidence is insufficient to establish the reliability and validity of cognitive assessments used in this way. We conclude that current evidence does not justify routine cognitive assessment for LD identification. As an alternative, we offer an instructional conceptualization of LD: a hybrid model that directly informs intervention and is based on documenting low academic achievement, inadequate response to intensive interventions, and a consideration of exclusionary factors.

Utility of KTEA-3 Error Analysis for the Diagnosis of Specific Learning Disabilities

Through the use of excerpts from one of our own case studies, this commentary applied concepts inherent in, but not limited to, the neuropsychological literature to the interpretation of performance on the Kaufman Tests of Educational Achievement–Third Edition (KTEA-3), particularly at the level of error analysis. The approach to KTEA-3 test interpretation advocated here parallels the cognitive process-oriented approach used by McCloskey and colleagues in their interpretation of the Wechsler scales. This approach is also advocated by Hale and Fiorello as part of their cognitive hypothesis testing model and is inherent in the neuropsychological assessment and interpretation frameworks proposed by Miller and Dehn. For the purpose of this commentary, we describe how this approach to KTEA-3 test interpretation fits within our own Cattell-Horn-Carroll (CHC)-based approach to specific learning disabilities (SLD) identification. To derive maximum benefit from error analysis, practitioners must pay careful attention to the manner in which students respond to test items and copiously document their observations during test administration.

Using Simulations to Investigate the Longitudinal Stability of Alternative Schemes for Classifying and Identifying Children with Reading Disabilities

The present study employed data simulation techniques to investigate the one-year stability of alternative classification schemes for identifying children with reading disabilities. Classification schemes investigated include low performance, unexpected low performance, dual-discrepancy, and a rudimentary form of constellation model of reading disabilities that included multiple criteria. Data from Spencer et al. (2014) were used to construct a growth model of reading development. The parameters estimated from this model were then used to construct three simulated datasets wherein the growth parameters were manipulated in one of three ways: A stable-growth pattern, a mastery learning pattern and a fan-spread pattern. Results indicated that overall the constellation model provided the most stable classifications across all conditions of the simulation, and that classification schemes were most stable in the fan-spread condition, and were the least stable under the mastery learning growth pattern. These results also demonstrate the utility of data simulations in reading research.

Neuropsychological Profiles of Written Expression Learning Disabilities Determined by Concordance-Discordance Model Criteria

Children with specific learning disabilities (SLD) have disparate neuropsychological processing deficits that interfere with academic achievement in spelling, writing fluency, and/or written expression (WE). Although there are multiple potential causes of WE SLD, there is a paucity of research exploring this critical academic skill from a neuropsychological perspective. This study examined the neuropsychological profiles of WE SLD subtypes defined using the concordance-discordance model (C-DM) of SLD identification. Participants were drawn from a sample of 283 children (194 boys, 89 girls) aged 6 years to 16 years old (M(age) = 9.58 years, SD = 2.29 years) referred for comprehensive neuropsychological evaluations in school settings and subsequently selected based on C-DM determined spelling, writing fluency, and WE SLD. WE SLD subtypes differed on several psychomotor, memory, and executive function measures (F range = 2.48-5.07, p range = .049 to <.001), suggesting that these children exhibit distinct patterns of neuropsychological processing strengths and weaknesses. Findings have relevance for differential diagnosis of WE subtypes, discriminating WE SLD subtypes from low WE achievement, and developing differentiated evidence-based instruction and intervention for children with WE SLD. Limitations and future research will be addressed.

Incorporating RTI in a Hybrid Model of Reading Disability

The present study seeks to evaluate a hybrid model of identification that incorporates response-to-intervention (RTI) as a one of the key symptoms of reading disability. The one-year stability of alternative operational definitions of reading disability was examined in a large scale sample of students who were followed longitudinally from first to second grade. The results confirmed previous findings of limited stability for single-criterion based operational definitions of reading disability. However, substantially greater stability was obtained for a hybrid model of reading disability that incorporates RTI with other common symptoms of reading disability.

Classification and identification of reading and math disabilities: the special case of comorbidity

Much of learning disabilities research relies on categorical classification frameworks that use psychometric tests and cut points to identify children with reading or math difficulties. However, there is increasing evidence that the attributes of reading and math learning disabilities are dimensional, representing correlated continua of severity. We discuss issues related to categorical and dimensional approaches to reading and math disabilities, and their comorbid associations, highlighting problems with the use of cut points and correlated assessments. Two simulations are provided in which the correlational structure of a set of cognitive and achievement data are simulated from a single population with no categorical structures. The simulations produce profiles remarkably similar to reported profile differences, suggesting that the patterns are a product of the cut point and the correlational structure of the data. If dimensional approaches better fit the attributes of learning disability, new conceptualizations and better methods to identification and intervention may emerge, especially for comorbid associations of reading and math difficulties.

Comorbidity of LD and ADHD: implications of DSM-5 for assessment and treatment

Attention-deficit/hyperactivity disorder (ADHD) and learning disability (LD) can co-occur for a significant minority of children with each disorder. A total of 17 studies (2001-2011) examining ADHD-LD comorbidity were reviewed, revealing a higher mean comorbidity rate (45.1%) than has been obtained previously. Higher comorbidity may be the result of including students with writing disorders, not just reading and/or math disabilities. Proposed DSM-5 criteria for both disorders will likely affect comorbidity rates; however, it is unclear whether such rates will increase or decrease. Regardless of the specific impact of DSM revisions, academic skill and/or performance deficits should be assessed for students with ADHD as part of screening, comprehensive evaluation, and treatment monitoring. Comprehensive intervention services for students with comorbid ADHD and LD will require empirically supported treatment strategies that address both disorders and that are implemented across school and home settings.