Standardizing procedures for calculating rorschach interrater reliability: conceptual and empirical foundations

Rorschach Performance Assessment System (R-PAS) Interrater Reliability in a Brazilian Adolescent Sample and Comparisons With Three Other Studies

We examine interrater reliability for scoring the Rorschach Performance Assessment System (R-PAS) in a sample of 89 adolescents (M_Age = 13.2, SD = 1.01) from Brazil using exact agreement intraclass correlations coefficient (ICCs) for the 60 protocol-level scores that are the focus of interpretation. The first author completed or reviewed all of the primary coding, and seven R-PAS proficient psychologists trained at different sites independently produced secondary coding. Overall, excellent agreement was found (M ICC = 0.89; SD = 0.09). When averaged across this study and three other comparison studies, stronger reliability was present, in general, for commonly coded variables (M = 0.87) as opposed to rare or infrequent variables (M = 0.78). In addition, 78.3% of the variables showed excellent interrater reliability and an additional 20.0% had good reliability. The results also showed that the ICCs for most variables had low variability across studies, suggesting clear coding guidelines. However, variables with higher ICC variability across studies indicated domains where it would be desirable to expand guidelines with more detailed parameters. Overall, the findings indicate excellent interrater reliability for the great majority of codes and present solid grounds for future research on interrater reliability with R-PAS.

Field Reliability of Comprehensive System Scoring in an Adolescent Inpatient Sample

The extent to which the Comprehensive System for the Rorschach is reliably scored has been a topic of some controversy. Although several studies have concluded it can be scored reliably in research settings, little is known about its reliability in field settings. This study evaluated the reliability of both response-level codes and protocol-level scores among 84 adolescent psychiatric inpatients in a clinical setting. Rorschachs were originally administered and scored for clinical purposes. Among response codes, 87% demonstrated acceptable reliability(> .60), and most coefficients exceeded .80. Results were similar for protocol-level scores, with only one score demonstrating less than adequate reliability. The findings are consistent with previous evidence, indicating reliable scoring is possible even in field settings.

Rorschach intercoder reliability for protocol-level comprehensive system variables in an international sample

This study examines the intercoder reliability of Rorschach Comprehensive System (CS; Exner, 2001) protocol-level variables. A large international sample was combined to obtain intercoder agreement for 489 Rorschach protocols coded using the CS. Intercoder agreement was calculated using an Iota coefficient, a statistical coefficient similar to kappa that is corrected for chance. Iota values for the variables analyzed ranged from .31 to 1.00, with 2 in the poor range of agreement, 4 in the fair range, 25 in the good range, and 116 in the excellent range of agreement. Discrepancies between variables are discussed.

Depression and Vulnerability As Assessed by the Rorschach Method

We examined clinically depressed (CD; n = 16), previously depressed (PD; n = 19) and never depressed (ND; n = 18) individuals on 13 theoretically selected Rorschach (Exner, 1993; Rorschach, 1942) variables and on the Beck Depression Inventory (BDI; Beck, Rush, Shaw, & Emery, 1979). The group assignment was made according to the criteria of Diagnostic and Statistical Manual of Mental Disorders (4th ed.; American Psychiatric Association, 1994). We tested 2 contradictory models for depressive vulnerability, Beck's (Clark & Beck, 1999) and Miranda and Persons's (1988; Persons & Miranda, 1992), in a planned comparison design with focused contrasts. The CDs significantly contrasted the combined group of NDs and the PDs in a pathological direction on 8 of the 13 Rorschach variables and on the BDI. However, the combined group of CDs and PDs also significantly contrasted the NDs in a pathological direction on 3 of these Rorschach variables and on the BDI. In addition, logistic regression analyses indicated that Rorschach indexes significantly improved the prediction of major depression above and beyond that achieved by the BDI. The findings show that the Rorschach method was able to identify (a) cognitive and aggressive disturbances that are present in individuals who are actively depressed but not in individuals who have been depressed in the past or never been depressed and (b) affective and coping disturbances that are present in depressed individuals and to some degree in PD individuals but not in individuals who have not experienced depression. We discuss the scanty evidence of psychological disturbances in PD individuals, as measured with the Rorschach, in relation to the mood-state dependent hypothesis of Miranda and Persons (1988; Persons & Miranda, 1992).

The mangled butterfly: Rorschach results from 45 violent psychopaths

Participants were 45 violent California male prison inmates scoring 30 or more on the Psychopathy Checklist-Revised (PCL-R; Hare, 1991, 2003). Inmates were evaluated using Rorschach and neuropsychological test data. The participants' intellectual functioning was within the low-average range and displayed a lack of flexibility. Rorschach data were not suggestive of chronic narcissism and anger as in other psychopathic samples. This group resembled Exner's normative sample of high Lambda adults. Consistent with previous studies, psychopaths demonstrated poor emotional modulation, diminished reality testing, little interest in people, and virtually no attachment capacity. Most utilized a simplistic, avoidant, and concrete style. This appeared to be consistent with the concrete thinking and fragmentation attributed to the criminal personality. Concrete thinking is based upon literal interpretations of events. Fragmentation is associated with attitudes that are situation specific and self-serving.

A Psychometric Evaluation of the Rorschach Comprehensive System's Perceptual Thinking Index

In this study, we investigated evidence for reliability and validity of the Perceptual Thinking Index (PTI; Exner, 2000a, 2000b) among an adult inpatient population. We conducted reliability and validity analyses on 107 patients who met the Diagnostic and Statistical Manual of Mental Disorders (4th ed., text revision; American Psychiatric Association, 2000) criteria for a schizophrenia-spectrum disorder (SSD) or mood disorder with no psychotic features (MD). Results provided support for interrater reliability as well as internal consistency of the PTI. Furthermore, the PTI was an effective index in differentiating SSD patients from patients diagnosed with an MD. Finally, the PTI demonstrated adequate diagnostic statistics that can be useful in the classification of patients diagnosed with SSD and MD. We discuss methodological issues, implications for assessment practice, and directions for future research.

Adolescentes com Lúpus Eritematoso Sistêmico: um estudo por meio do método de rorschach

Características da personalidade de adolescentes com Lúpus Eritematoso sistêmico (LES) foram estudadas por meio do método de Rorschach e da Escala Wechsler de Inteligência. A atividade da doença foi avaliada através do Índice de Atividade do LES (SLEDAI). Na Escala Wechsler, o QI médio do Grupo Índice foi menor que o do Grupo Controle. No Rorschach, as pacientes com lúpus apresentaram maior dificuldade nas relações interpessoais e na auto-estima, porém com recursos para processar afeto e tolerar estresse. Foi observada correlação positiva moderada entre o índice da atividade da doença e a proporção de constrição afetiva: quanto maior o escore do SLEDAI, menor a capacidade para processar emoção. A correlação entre o SLEDAI e o QI foi positiva: quanto maior o índice de atividade do LES, menos recursos intelectuais se mostram disponíveis.

Prediction of adolescent and adult delinquency from childhood Rorschach ratings

Relatively few studies have predicted adolescent or adult delinquency from childhood Rorschach data. Incremental validity of Rorschach measures over other known consistent and more easily obtained predictors has typically not been investigated. We addressed Rorschach data's incremental validity over mother's reports of externalizing behavior and mother-child relations for predicting officially registered adolescent and adult delinquency in a longitudinal study of 122 Swedish men. The Rorschach measures we used were aggregates of blind, prospectively performed, global expert ratings based on a holistic evaluation of protocols. An aggregate of Rorschach-based ratings of ego strength significantly improved the prediction of delinquency in adolescence and adulthood over and above earlier identified best predictors (mother's ratings of mother-child relations and externalizing problem behavior).

Temporal stability in the Rorschach method: a meta-analytic review

The temporal stabilities of the Rorschach method scoring systems were investigated. A comprehensive literature search was conducted to find all test-retest studies, both regular and from control samples in therapy evaluation studies. Meta-analyses were conducted for samples, individual entries, and variables in 36 samples. Separate analyses were made for 10 samples using the Comprehensive System (CS; Exner, 1993). The temporal stability was high, especially those obtained from regression model predictions. The CS consistently shows higher stability than other systems, whereas increasing retest intervals leads to decreasing stability. Shortcomings of the available studies are highlighted and discussed, and recommendations are given for design and methodology.

Rorschach administration: a comparison of the effect of two instructions given to an inpatient sample of drug addicts

The effect of administering the Rorschach Inkblot Method under two instructional sets was compared on three classes of outcome variables: the frequency with which subjects asked questions about the test; the frequency of brief protocols (fewer than 14 responses); and 17 traditional Rorschach structural summary scores. Sixty subjects, obtained from three inpatient psychiatric clinics treating drug addicts, randomly received either the short pre-testing instruction "What might this be?" originally developed by Herman Rorschach and recommended in the Comprehensive System, or a longer and more elaborated instruction, which for many years has been the standard instruction in Norway. Compared with the Norwegian instruction, the short instruction produced significantly more questions to the examiner about the test. For the other outcome measures no differences were observed.

Psychological measures as predictors of military training performance

The predictive validity of 7 ability tests, the Big Five, and the Rorschach method administered to 71 male applicants at the Naval Special Forces (NSF) of Norway was evaluated based on pass/fail results in training. The findings showed: (a) small correlations between the ability tests, the Big Five scales, and the success criterion; (b) Rorschach variables measuring stress tolerance, reality testing, cognition, and social adjustment correlated significantly (r =.25 to.48) with pass/fail results in training, and (c) logistic regression analysis revealed that 3 of the Rorschach variables accumulated incrementally in the prediction of training completion when entered after the ability tests and the Big Five scales, thus supporting the merit of using Rorschach variables for predicting NSF training performance

Empirical support for interrater reliability of Rorschach Comprehensive System coding

Although a great deal of data has been published in the past 20 years supporting the interrater reliability of the Rorschach, recently commentators have raised anew concerns over the interrater reliability of this well-known and frequently used measure. An analysis of the literature reveals that these concerns are based on a selective review of the literature. The current study reports interrater reliability statistics for 70 interpretive Rorschach variables (n = 84), breaking down intraclass correlations statistics by base rate. Results confirm the strong empirical evidence of the interrater reliability of this measure when scored by a well-trained and diverse group of researchers and clinicians. Reliability is especially strong and consistent for the high base-rate variables from which clinicians often base their interpretations. These data further suggest that large samples are needed for stable reliability estimates of low base-rate variables and that estimates of the reliability of low base-rate variables are subject to error.

Estudo normativo do sistema compreensivo do Rorschach para a cidade de São Paulo

O método de Rorschach tem se mostrado bastante eficiente como instrumento na área de investigação e diagnóstico da personalidade. O sistema compreensivo de Exner tem garantido ao Rorschach status de instrumento de grande confiabilidade, por sua objetividade e precisão nos resultados. Entretanto, para que isso ocorra, é necessário trabalhar com normas apropriadas para a população. Por isso, realizamos uma pesquisa de normatização a fim de encontrar resultados para melhor utilização do sistema compreensivo em nosso meio. Aplicamos o Rorschach individualmente, em 200 sujeitos não-pacientes, de ambos os sexos, residentes na cidade de São Paulo, com idade variando entre 17 e 65 anos, de todos os graus de escolaridade e níveis socioeconômicos. Os resultados apresentam várias diferenças, quando comparados às normas apresentadas por Exner, mas algumas semelhanças com os estudos apresentados por especialistas de outros países, especialmente, os de origem latina. Estes resultados indicam influências culturais em diversas variáveis do Rorschach.

An examination of interrater reliability for scoring the Rorschach Comprehensive System in eight data sets

In this article, we describe interrater reliability for the Comprehensive System (CS; Exner. 1993) in 8 relatively large samples, including (a) students, (b) experienced re- searchers, (c) clinicians, (d) clinicians and then researchers, (e) a composite clinical sample (i.e., a to d), and 3 samples in which randomly generated erroneous scores were substituted for (f) 10%, (g) 20%, or (h) 30% of the original responses. Across samples, 133 to 143 statistically stable CS scores had excellent reliability, with median intraclass correlations of.85, .96, .97, .95, .93, .95, .89, and .82, respectively. We also demonstrate reliability findings from this study closely match the results derived from a synthesis of prior research, CS summary scores are more reliable than scores assigned to individual responses, small samples are more likely to generate unstable and lower reliability estimates, and Meyer's (1997a) procedures for estimating response segment reliability were accurate. The CS can be scored reliably, but because scoring is the result of coder skills clinicians must conscientiously monitor their accuracy.

Interobserver agreement, intraobserver reliability, and the Rorschach Comprehensive System

Interrater agreement and reliability for the Rorschach have recently come under increasing scrutiny. This is the second report examining methods of Comprehensive System reliability using principles derived from observational methodology and applied behavioral analysis. This study examined a previous nonpatient sample of 20 protocols (N = 412 responses) and also examined a new clinical sample of 20 protocols (N = 374 responses) diagnosed with Research Diagnostic Criteria. Reliability was analyzed at multiple levels of Comprehensive System data, including response-level individual codes and coding decisions and ratios, percentages, and derivations from the Structural Summary. With a number of exceptions, most Comprehensive System codes, coding decisions, and summary scores yield acceptable, and in many instances excellent, levels of reliability. Limitations arising from the nature of Rorschach data and Comprehensive System coding criteria are discussed.

The Rorschach Inkblot Test: a case of overstatement?

In the 1940s, inflated claims were often made regarding the Rorschach Inkblot Test. Over half a century later, overstatements regarding the test are still common. The present article identifies problems with the Rorschach regarding norms, cultural sensitivity, interrater reliability, test-retest reliability, validity, factor structure, and accessibility of supporting studies. Contrary to overstated claims made on behalf of the Rorschach, the test continues to be a highly problematic instrument from a psychometric standpoint.

Behavioral science foundations of the Rorschach test: research and clinical applications

Never without its critics, the Rorschach Test continues to be widely used in clinical settings. The test continues to be criticized vigorously. Rorschach critics appear to fall into two broad groups: those leveling valid methodological concerns about the test s behavioral science foundations and method critics who appear to deny the validity of the test on strictly a priori or theoretical considerations. Many critics do not appear to be acquainted with the extensive Rorschach research literature. The current paper provides an overview of several domains of applied and laboratory Rorschach behavioral science, including statistical power analysis, interobserver agreement and interrater reliability, Rorschach assessment of thought disorder, and emerging research linking Rorschach variables with diagnostic criteria from the DSM-IV, as a means of educating both adherents and detractors alike concerning the test s scientific track record and applicability to clinical assessment.

Methodological issues in evaluating Rorschach validity: a comment on Burns and Viglione (1996), Weiner (1996), and Ganellen (1996)

The old controversy regarding the Rorschach Inkblot Test has recently revived. The present article suggests that the debate will be most productive if careful attention is paid to methodological issues. Three recent examples illustrate how incorrect conclusions regarding Rorschach validity may occur if methodological issues are not evaluated carefully. The present article examines (a) Burns and Viglione s (1996) conclusion that the Rorschach Human Experience Variable (HEV) is a predictor of interpersonal relatedness among adults; (b) Weiner s (1996) conclusion that the D score and Morbid Responses (MOR) are valid measures of experienced distress in Posttraumatic Stress Disorder (PTSD); and (c) Ganellen s (1996a, 1996b) conclusion that the Rorschach Depression Index (DEPI) and the Minnesota Multiphasic Personality Inventory (MMPI) are comparable in their power to identify diagnoses of depression.

Contemporary Perspectives on Rorschach Assessment

Summary: In the last 20 years the advent of the Comprehensive System has developed the Rorschach Inkblot Method (RIM) into a standardized and psychometrically sound personality assessment instrument with numerous indices that can be reliably coded, show considerable test-retest stability, and have abundant valid corollaries. Rorschach assessment has demonstrated utility in contributing to clinical diagnosis of psychopathology, providing guidelines for treatment planning and outcome evaluation, and identifying adaptive and maladaptive features of how people attend to their experience, use ideation, modulate affect, manage stress, view themselves, and relate to others. The RIM can also be effectively integrated with more structured self-report inventories in a complementary way that describes personality functioning in greater depth than would otherwise be possible. The RIM remains at present a widely used and extensively researched measure in many parts of the world. Although the inkblot method is essentially a culture-free instrument with universal applicability, further research is needed to establish cross-cultural normative data concerning its variables.

Two Foundations of Rorschach Assessment Revisited

Summary: The Rorschach has been approached from two philosophical traditions: the hermeneutic and the scientific. The dialectical tension between these diverse traditions has led to considerable argument about the professional status of the Rorschach. Literature emanating from these traditions is reviewed and conclusions reached about the reliability and validity of the instrument from both approaches. Arguments whether the approaches are mutually exclusive and whether the approaches can be mixed are examined. A research program is recommended that follows Meehl's (1990) discussion of a Lakatosian approach to scientific theory testing.

The Rorschach Schizophrenia Index (SCZI): an examination of reliability, validity, and diagnostic efficiency

In this study, we investigate the reliability, validity, and diagnostic efficiency of the Rorschach Schizophrenia Index (SCZI) in relation to the accurate identification of patients diagnosed with Diagnostic and Statistical Manual of Mental Disorders (4th ed. [DSM-IV], American Psychiatric Association, 1994) schizophrenia or other psychotic disorder (PD) according to the methodological recommendations offered by Wood, Nezworski, and Stejskal (1996). Seventy-eight patients who were found to meet DSM-IV criteria for a PD or Axis II disorder (PD = 33; borderline personality disorder = 23; Cluster A personality disorders = 9; Cluster C personality disorders = 13) and 50 nonclinical participants were compared on the SCZI. The results of this study indicate that the SCZI is internally consistent and can be reliably scored. In addition, the SCZI was used effectively in differentiating PD patients from patients with an Axis II disorder and from the participants in the nonclinical sample. Also, the SCZI variable was found to be empirically related to the presence of a DSM-IV diagnosis of PD. Finally, this variable could be employed for classification purposes in ways that were clinically meaningful in the diagnosis of a PD. Conceptual and methodological issues are discussed in relation to the assessment of psychosis.