Comparison of Assessment of Citrus Canker Foliar Symptoms by Experienced and Inexperienced Raters

Standard Area Diagrams for Pecan Leaf Scab: Effect of Rater Experience, Location, and Leaf Size on Reliability and Accuracy of Visual Estimates

Assessments of the severity of scab (Venturia effusa), an economically significant disease of pecan, are critical for determining pecan cultivar susceptibility, disease epidemiology, and integrated disease management approaches. We developed a standard area diagram (SAD) set to aid in assessments of pecan leaflet scab. Leaflets with scab lesions were harvested and scanned using a flatbed scanner at 600 dpi, and Fiji (ImageJ) was used to determine the actual percent disease severity. The SADs had 10 leaflets ranging in severity from 0.2 to 48.9%. Forty "small" (1.34 to 7.43 cm2) and 40 "large" (7.67 to 25.9 cm2) leaflet images were randomized for rater assessments. The images were assessed twice by 36 raters, first without and then with the SADs as a guide. Data were subjected to analysis using Lin's concordance correlation coefficient (LCC, pc) to determine the accuracy of ratings and by intraclass correlation coefficient (ICC) analysis to determine interrater reliability. The effects of rater experience, rater location, and leaflet size were also determined. The SADs significantly improved the agreement between raters and the actual values (LCC, pc = 0.70 and 0.84 without and with the SADs, respectively). The reliability of estimates was improved (ICC = 0.54 and 0.82 without and with the SADs, respectively). The effect of rater location on overall concordance was significant without and with the SADs based on an analysis of variance using a generalized linear model and lsmeans separation (P < 0.05). A generalized linear mixed model analysis revealed that there was a significant interaction between rater location, experience, and the use of the SADs, with some raters having greater improvement in generalized bias and concordance. Raters had a significantly better accuracy when rating "small" leaves (LCC, pc = 0.86) compared with "large" leaves (LCC, pc = 0.82) when using the SADs, highlighting the impact of psychophysics on field evaluations of plant disease severity. The proposed SADs will serve as an improved tool for performing pecan leaflet scab assessments by the pecan research community.

Visual Assessment of Phony Peach Disease: Evaluating Rater Accuracy and Reliability

Phony peach disease (PPD), found predominantly in central and southern Georgia, is a re-emerging disease caused by Xylella fastidiosa (Xf) subsp. multiplex. Accurate detection and rapid removal of symptomatic trees are crucial to effective disease management. Currently, peach producers rely solely on visual identification of symptoms to confirm PPD, which can be ambiguous if early in development. We compared visual assessment to quantitative PCR (qPCR) for detecting Xf in 'Julyprince' in 2019 and 2020 (JP2019 and JP2020) and in 'Scarletprince' in 2020 (SP2020). With no prior knowledge of qPCR results, all trees in each orchard were assessed by a cohort of five experienced and five inexperienced raters in the morning and afternoon. Visual identification accuracy of PPD was variable, but experienced raters were more accurate when identifying PPD trees. In JP2019, the mean rater accuracy for experienced and inexperienced raters was 0.882 and 0.805, respectively. For JP2020, the mean rater accuracy for experienced and inexperienced raters was 0.914 and 0.816, respectively. For SP2020, the mean rater accuracy for experienced and inexperienced raters was 0.898 and 0.807, respectively. All raters had false positive (FP) and false negative (FN) observations, but experienced raters had significantly lower FN rates compared with the inexperienced group. Almost all raters overestimated the incidence of PPD in the orchards. Reliability of visual assessments was demonstrated as moderate to good, regardless of experience. Further research is needed to develop accurate and reliable methods of detection to aid management of PPD as both FPs and FNs are costly to peach production.

Survival Analysis as a Basis for Testing Hypotheses when Using Quantitative Ordinal Scale Disease Severity Data

Disease severity in plant pathology is often measured by the amount of a plant or plant part that exhibits disease symptoms. This is typically assessed using a numerical scale, which allows a standardized, convenient, and quick method of rating. These scales, known as quantitative ordinal scales (QOS), divide the percentage scale into a predetermined number of intervals. There are various ways to analyze these ordinal data, with traditional methods involving the use of midpoint conversion to represent the interval. However, this may not be precise enough, as it is only an estimate of the true value. In this case, the data may be considered interval-censored, meaning that we have some knowledge of the value but not an exact measurement. This type of uncertainty is known as censoring, and techniques that address censoring, such as survival analysis (SA), use all available information and account for this uncertainty. To investigate the pros and cons of using SA with QOS measurements, we conducted a simulation based on three pathosystems. The results showed that SA almost always outperformed midpoint conversion with data analyzed using a t test, particularly when data were not normally distributed. Midpoint conversion is currently a standard procedure. In certain cases, the midpoint approach required a 400% increase in sample size to achieve the same power as the SA method. However, as the mean severity increases, fewer additional samples are needed (approximately an additional 100%), regardless of the assessment method used. Based on these findings, we conclude that SA is a valuable method for enhancing the power of hypothesis testing when analyzing QOS severity data. Future research should investigate the wider use of survival analysis techniques in plant pathology and their potential applications in the discipline.

Validation of Standard Area Diagrams to Estimate the Severity of Septoria Leaf Spot on Stevia in Paraguay, Mexico, and the United States

Septoria leaf spot (SLS) affects stevia leaves, reducing their quality. Estimates of SLS severity on different genotypes are made to identify resistance and as a basis to compare management approaches. The use of standard area diagrams (SADs) can improve the accuracy and reliability of severity estimates. In this study, we developed new SADs with six illustrations (0.5, 1, 10, 25, 40, and 75% severity). The SADs were validated by raters with and without experience in estimating SLS. Raters evaluated 40 leaf photos with SLS severities ranging from 0 to 100% without and with the SADs. Agreement (ρ_c), bias (C_b), precision (r), and intracluster correlation (ρ) coefficients were significantly closer to "true" severity values when the SADs was used by inexperienced (ρ_c = 0.89; C_b = 0.97; r = 0.90, ρ = 0.81) and experienced (ρ_c = 0.94; C_b = 0.99; r = 0.95, ρ = 0.91) raters. The SADs were tested under field conditions in Paraguay, Mexico, and the United States, with inexperienced raters assigned to two groups, one SADs trained and the other not trained, that estimated SLS severity three times: first, all raters without SADs and no time limit for the estimates; second, only the SADs-trained group used SADs and no time limit; and third, only the SADs-trained group used SADs, with a time limit of 10 s imposed per specimen assessment. Agreement and reliability of SLS severity estimates significantly improved when raters used the SADs without a time limit. The use of the new SADs improved the accuracy, precision, and reliability of SLS severity estimates, enhancing the uniformity in assessment across different stevia programs.

Oat Crown Rust Disease Severity Estimated at Many Time Points Using Multispectral Aerial Photos

All plant breeding programs are dependent on plant phenotypic and genotypic data, but the development of phenotyping technology has been slow relative to that of genotyping. Crown rust (Puccinia coronata f. sp. avenae Erikss.) is the most important disease of cultivated oat (Avena sativa L.), making the development of disease-resistant oat cultivars an important breeding objective. Visual observation is the most common scoring method, but it can be laborious and subjective. We visually scored a diverse collection of 256 oat lines at a total of 27 time points in three disease nursery environments. Multispectral aerial photos were collected using an unmanned aerial vehicle at the same time points as the visual observations. The photos were analyzed, and subsets of the spectral properties of each plot were measured. Random forest modeling was used to model the relationship between the spectral properties of the plots and visually observed disease severity. The ability of the photo data and the random forest model to estimate visually observed disease severity was evaluated using three different cross-validation analyses. We specifically addressed the issue of assessing phenotyping accuracy across and within time points. The accuracy of the photo estimates was greatest for adult plants shortly before they began to senesce. Accuracy outside of that time frame was generally low but statistically significant. Unmanned aerial vehicle-mounted sensors could increase disease scoring efficiency, but additional investigation into the spectral signature of disease severity at all plant growth stages may be necessary to automate accurate full-season measurements.

Understanding the ramifications of quantitative ordinal scales on accuracy of estimates of disease severity and data analysis in plant pathology

The severity of plant diseases, traditionally defined as the proportion of the plant tissue exhibiting symptoms, is a key quantitative variable to know for many diseases but is prone to error. Plant pathologists face many situations in which the measurement by nearest percent estimates (NPEs) of disease severity is time-consuming or impractical. Moreover, rater NPEs of disease severity are notoriously variable. Therefore, NPEs of disease may be of questionable value if severity cannot be determined accurately and reliably. In such situations, researchers have often used a quantitative ordinal scale of measurement-often alleging the time saved, and the ease with which the scale can be learned. Because quantitative ordinal disease scales lack the resolution of the 0 to 100% scale, they are inherently less accurate. We contend that scale design and structure have ramifications for the resulting analysis of data from the ordinal scale data. To minimize inaccuracy and ensure that there is equivalent statistical power when using quantitative ordinal scale data, design of the scales can be optimized for use in the discipline of plant pathology. In this review, we focus on the nature of quantitative ordinal scales used in plant disease assessment. Subsequently, their application and effects will be discussed. Finally, we will review how to optimize quantitative ordinal scales design to allow sufficient accuracy of estimation while maximizing power for hypothesis testing.

Importance of Molecular Data to Identify Fungal Plant Pathogens and Guidelines for Pathogenicity Testing Based on Koch's Postulates

Fungi are an essential component of any ecosystem, but they can also cause mild and severe plant diseases. Plant diseases are caused by a wide array of fungal groups that affect a diverse range of hosts with different tissue specificities. Fungi were previously named based only on morphology and, in many cases, host association, which has led to superfluous species names and synonyms. Morphology-based identification represents an important method for genus level identification and molecular data are important to accurately identify species. Accurate identification of fungal pathogens is vital as the scientific name links the knowledge concerning a species including the biology, host range, distribution, and potential risk of the pathogen, which are vital for effective control measures. Thus, in the modern era, a polyphasic approach is recommended when identifying fungal pathogens. It is also important to determine if the organism is capable of causing host damage, which usually relies on the application of Koch's postulates for fungal plant pathogens. The importance and the challenges of applying Koch's postulates are discussed. Bradford Hill criteria, which are generally used in establishing the cause of human disease, are briefly introduced. We provide guidelines for pathogenicity testing based on the implementation of modified Koch's postulates incorporating biological gradient, consistency, and plausibility criteria from Bradford Hill. We provide a set of protocols for fungal pathogenicity testing along with a severity score guide, which takes into consideration the depth of lesions. The application of a standard protocol for fungal pathogenicity testing and disease assessment in plants will enable inter-studies comparison, thus improving accuracy. When introducing novel plant pathogenic fungal species without proving the taxon is the causal agent using Koch's postulates, we advise the use of the term associated with the "disease symptoms" of "the host plant". Where possible, details of disease symptoms should be clearly articulated.

Reproducibility of the Development and Validation Process of Standard Area Diagram by Two Laboratories: An Example Using the Botrytis cinerea/Gerbera jamesonii Pathosystem

Standard area diagrams (SADs) are plant disease severity assessment aids demonstrated to improve the accuracy and reliability of visual estimates of severity. Knowledge of the sources of variation, including those specific to a lab such as raters, specific procedures followed including instruction, image analysis software, image viewing time, etc., that affect the outcome of development and validation of SADs can help improve standard operating practice of these assessment aids. As reproducibility has not previously been explored in development of SADs, we aimed to explore the overarching question of whether the lab in which the measurement and validation of a SAD was performed affected the outcome of the process. Two different labs (Lab 1 and Lab 2) measured severity on the individual diagrams in a SAD and validated them independently for severity of gray mold (caused by Botrytis cinerea) on Gerbera daisy. Severity measurements of the 30 test images were performed independently at the two labs as well. A different group of 18 raters at each lab assessed the test images first without, and secondly with SADs under independent instruction at both Lab 1 and 2. Results showed that actual severity on the SADs as measured at each lab varied by up to 5.18%. Furthermore, measurement of the test image actual values varied from 0 to up to 24.29%, depending on image. Whereas at Lab 1 an equivalence test indicated no significant improvement in any measure of agreement with use of the SADs, at Lab 2, scale shift, generalized bias, and agreement were significantly improved with use of the SADs (P ≤ 0.05). An analysis of variance indicated differences existed between labs, use of the SADs aid, and the interaction, depending on the agreement statistic. Based on an equivalence test, the interrater reliability was significantly (P ≤ 0.05) improved at both Lab 1 and Lab 2 as a result of using SADs as an aid to severity estimation. Gain in measures of agreement and reliability tended to be greatest for the least able raters at both Lab 1 and Lab 2. Absolute error was reduced at both labs when raters used SADs. The results confirm that SADs are a useful tool, but the results demonstrated that aspects of the development and validation process in different labs may affect the outcome.

Quantitative Ordinal Scale Estimates of Plant Disease Severity: Comparing Treatments Using a Proportional Odds Model

Studies in plant pathology, agronomy, and plant breeding requiring disease severity assessment often use quantitative ordinal scales (i.e., a special type of ordinal scale that uses defined numeric ranges); a frequently used example of such a scale is the Horsfall-Barratt scale. Parametric proportional odds models (POMs) may be used to analyze the ratings obtained from quantitative ordinal scales directly, without converting ratings to percent area affected using range midpoints of such scales (currently a standard procedure). Our aim was to evaluate the performance of the POM for comparing treatments using ordinal estimates of disease severity relative to two alternatives, the midpoint conversions (MCs) and nearest percent estimates (NPEs). A simulation method was implemented and the parameters of the simulation estimated using actual disease severity data from the field. The criterion for comparison of the three approaches was the power of the hypothesis test (the probability to reject the null hypothesis when it is false). Most often, NPEs had superior performance. The performance of the POM was never inferior to using the MC at severity <40%. Especially at low disease severity (≤10%), the POM was superior to using the MC method. Thus, for early onset of disease or for comparing treatments with severities <40%, the POM is preferable for analyzing disease severity data based on quantitative ordinal scales when comparing treatments and at severities >40% is equivalent to other methods.

Evaluation of App-Embedded Disease Scales for Aiding Visual Severity Estimation of Cercospora Leaf Spot of Table Beet

Two diagrammatic ordinal scales are available in the Estimate app (2017 version) for Cercospora leaf spot (CLS) severity on table beet: 10% linear (linear-based diagrammatic scale [LIN]) and logarithmic based (Horsfall-Barratt [HB]). These allow for estimating severity data of four types depending on the system used. A group of 30 raters assigned percentage severity on 30 photographs of diseased table beet leaves during five rounds first without an aid and then using each of the four rating systems in Estimate. In two, the perceived ordinal score of the HB or LIN scale was assigned where severity of the subject fit best. HB2 and LIN2 involved a second choice of unitary severity within the perceived score interval. There was large variation in unaided ability of raters to estimate severity: 13% were accurate (Lin's concordance correlation [LCC] > 0.9), 23% were inaccurate (LCC < 0.7), and the remaining had moderate accuracy. Larger disparities between assigned and actual ordinal scores (mostly overestimates) occurred using the LIN compared with the HB. The LIN2 produced the most accurate estimates (Lin's concordance correlation coefficient, ρ_c = 0.96; generalized bias parameter, C_b = 0.99; Pearson's correlation coefficient r = 0.95) and the greatest interrater reliability (overall concordance correlation coefficient and intraclass correlation coefficient > 0.93). The two-step process using the 10% linear scale is recommended for severity estimates of CLS in table beet.

Accuracy, reliability, and timing of visual evaluations of decay in fresh-cut lettuce

Visual assessments are used for evaluating the quality of food products, such as fresh-cut lettuce packaged in bags with modified atmosphere. We have compared the accuracy and the reliability of visual evaluations of decay on fresh-cut lettuce performed with experienced and inexperienced raters. In addition, we have analyzed decay data from over 4.5 thousand bags to determine the optimum timing for evaluations to detect differences among accessions. Lin's concordance coefficient (ρc) that takes into consideration both the closeness of the data and the conformance to the identity line showed high repeatability (intra-rater reliability, ρc = 0.97), reproducibility (inter-rater reliability, ρc = 0.92), and accuracy (ρc = 0.96) for experienced raters. Inexperienced raters did not perform as well and their ratings showed decreased repeatability (ρc = 0.93), but even larger reduction in reproducibility (ρc = 0.80) and accuracy (ρc = 0.90). We have detected that 5.3% of ratings were outside of the 95% limits of agreement. These under- or overestimates were predominantly found for bags with intermediate levels of decay, which corresponds to the middle of the rating scale. This occurs because intermediate amounts of decay are more difficult to discriminate than extremes. The frequencies of aberrant ratings for experienced raters ranged from 0.6% to 4.4% (mean = 2.1%), for inexperienced raters the frequencies were substantially higher, ranging from 6.1% to 15.6% (mean = 9.4%). Therefore, we recommend that new raters receive training that includes practical examples in this range of decay, use of standard area diagrams, and continuing interaction with experienced raters (consultation during actual rating). Very high agreement among experienced raters indicate that visual ratings can be successfully used for evaluations of decay, until a more objective, rapid, and affordable method is developed. We recommend evaluating samples at multiple time points until 42 days after processing (about 80% decay on average) and then combining these individual ratings into the area under the decay progress stairs (AUDePS) score. Applying this approach, experienced evaluators can accurately detect difference among lettuce accessions and identify lettuce cultivars with reduced decay.

Automated Identification of Northern Leaf Blight-Infected Maize Plants from Field Imagery Using Deep Learning

Northern leaf blight (NLB) can cause severe yield loss in maize; however, scouting large areas to accurately diagnose the disease is time consuming and difficult. We demonstrate a system capable of automatically identifying NLB lesions in field-acquired images of maize plants with high reliability. This approach uses a computational pipeline of convolutional neural networks (CNNs) that addresses the challenges of limited data and the myriad irregularities that appear in images of field-grown plants. Several CNNs were trained to classify small regions of images as containing NLB lesions or not; their predictions were combined into separate heat maps, then fed into a final CNN trained to classify the entire image as containing diseased plants or not. The system achieved 96.7% accuracy on test set images not used in training. We suggest that such systems mounted on aerial- or ground-based vehicles can help in automated high-throughput plant phenotyping, precision breeding for disease resistance, and reduced pesticide use through targeted application across a variety of plant and disease categories.

The Effect of Horsfall-Barratt Category Size on the Accuracy and Reliability of Estimates of Pecan Scab Severity

Pecan scab (Fusicladium effusum) is a destructive pecan disease. Disease assessments may be made using interval-scale-based methods or estimates of severity to the nearest percent area diseased. To explore the effects of rating method-Horsfall-Barratt (H-B) scale estimates versus nearest percent estimates (NPEs)-on the accuracy and reliability of severity estimates over different actual pecan scab severity ranges on fruit valves, raters assessed two cohorts of images with actual area (0 to 6, 6⁺ to 25%, and 25⁺ to 75%) diseased. Mean estimated disease within each actual disease severity range varied substantially. Means estimated by NPE within each actual disease severity range were not necessarily good predictors of the H-B scale estimate at <25% severity. H-B estimates by raters most often placed severity in the wrong category compared with actual disease. Measures of bias, accuracy, precision, and agreement using Lin's concordance correlation depended on the range of actual severity, with improvements increasing with actual disease severity category (from 0 to 6 through 25⁺ to 75%); however, the improvement was unaffected by the H-B assessments. Bootstrap analysis indicated that NPEs provided either equally good or more accurate and precise estimate of disease compared with the H-B scale at severities of 25⁺ to 75%. Inter-rater reliability using NPEs was greater at 25⁺ to 75% actual disease severity compared with using the H-B scale. Using NPEs compared with the H-B scale will more often result in more precise and accurate estimates of pecan scab severity, particularly when estimating actual disease severities of 25⁺ to 75%.

Pecan Scab Severity-Effects of Assessment Methods

Pecan scab is caused by the fungus Fusicladium effusum, and is the most destructive disease of pecan in the United States. Accurate and reliable disease assessments are needed to ensure that data provide a measure of actual disease intensity. The Horsfall-Barratt (H-B) category scale and its derivatives are commonly used to assess disease. Estimates using the H-B scale were compared with nearest percent estimate (NPEs) for rating disease severity of pecan scab on valves of fruit. Both inexperienced and experienced raters were included in the experiment. Lin's concordance correlation showed that agreement using NPEs was variable (ρ_c = 0.57 to 0.96), whereas estimates of disease severity using the H-B scale had similar agreement among most raters (ρ_c = 0.59 to 0.98). Converted values of NPEs to the H-B midpoints (NPEH-B) also provided a similar range (ρ_c = 0.61 to 0.96). Neither experienced nor inexperienced raters were consistently better using any of the three methods. Bootstrap analysis indicated that, among experienced raters, precision (r) and agreement (ρ_c) were often reduced when using the H-B scale compared with NPEs. There was no consistent effect of converting NPEs to NPEH-B midpoint values compared with actual H-B values. Inter-rater reliability using the H-B scale was never better than NPEs. Bootstrap analysis indicated no difference in the length of time needed to assess disease but regression analysis suggested that raters who were inherently fast in assessing disease with NPEs were often slower when using the H-B scale; conversely, raters who were slow assessing with NPEs were often faster when using the H-B scale. Thus, there appears to be no advantage in accuracy or reliability or reduction in time when inexperienced or experienced raters used a category rating scale to assess pecan scab.

In the eye of the beholder: the effect of rater variability and different rating scales on QTL mapping

The agronomic importance of developing durably resistant cultivars has led to substantial research in the field of quantitative disease resistance (QDR) and, in particular, mapping quantitative trait loci (QTL) for disease resistance. The assessment of QDR is typically conducted by visual estimation of disease severity, which raises concern over the accuracy and precision of visual estimates. Although previous studies have examined the factors affecting the accuracy and precision of visual disease assessment in relation to the true value of disease severity, the impact of this variability on the identification of disease resistance QTL has not been assessed. In this study, the effects of rater variability and rating scales on mapping QTL for northern leaf blight resistance in maize were evaluated in a recombinant inbred line population grown under field conditions. The population of 191 lines was evaluated by 22 different raters using a direct percentage estimate, a 0-to-9 ordinal rating scale, or both. It was found that more experienced raters had higher precision and that using a direct percentage estimation of diseased leaf area produced higher precision than using an ordinal scale. QTL mapping was then conducted using the disease estimates from each rater using stepwise general linear model selection (GLM) and inclusive composite interval mapping (ICIM). For GLM, the same QTL were largely found across raters, though some QTL were only identified by a subset of raters. The magnitudes of estimated allele effects at identified QTL varied drastically, sometimes by as much as threefold. ICIM produced highly consistent results across raters and for the different rating scales in identifying the location of QTL. We conclude that, despite variability between raters, the identification of QTL was largely consistent among raters, particularly when using ICIM. However, care should be taken in estimating QTL allele effects, because this was highly variable and rater dependent.

Some consequences of using the Horsfall-Barratt scale for hypothesis testing

Comparing treatment effects by hypothesis testing is a common practice in plant pathology. Nearest percent estimates (NPEs) of disease severity were compared with Horsfall-Barratt (H-B) scale data to explore whether there was an effect of assessment method on hypothesis testing. A simulation model based on field-collected data using leaves with disease severity of 0 to 60% was used; the relationship between NPEs and actual severity was linear, a hyperbolic function described the relationship between the standard deviation of the rater mean NPE and actual disease, and a lognormal distribution was assumed to describe the frequency of NPEs of specific actual disease severities by raters. Results of the simulation showed standard deviations of mean NPEs were consistently similar to the original rater standard deviation from the field-collected data; however, the standard deviations of the H-B scale data deviated from that of the original rater standard deviation, particularly at 20 to 50% severity, over which H-B scale grade intervals are widest; thus, it is over this range that differences in hypothesis testing are most likely to occur. To explore this, two normally distributed, hypothetical severity populations were compared using a t test with NPEs and H-B midpoint data. NPE data had a higher probability to reject the null hypothesis (H0) when H0 was false but greater sample size increased the probability to reject H0 for both methods, with the H-B scale data requiring up to a 50% greater sample size to attain the same probability to reject the H0 as NPEs when H0 was false. The increase in sample size resolves the increased sample variance caused by inaccurate individual estimates due to H-B scale midpoint scaling. As expected, various population characteristics influenced the probability to reject H0, including the difference between the two severity distribution means, their variability, and the ability of the raters. Inaccurate raters showed a similar probability to reject H0 when H0 was false using either assessment method but average and accurate raters had a greater probability to reject H0 when H0 was false using NPEs compared with H-B scale data. Accurate raters had, on average, better resolving power for estimating disease compared with that offered by the H-B scale and, therefore, the resulting sample variability was more representative of the population when sample size was limiting. Thus, there are various circumstances under which H-B scale data has a greater risk of failing to reject H0 when H0 is false (a type II error) compared with NPEs.

Automated Image Analysis of the Severity of Foliar Citrus Canker Symptoms

Citrus canker (caused by Xanthomonas citri subsp. citri) is a destructive disease, reducing yield and rendering fruit unfit for fresh sale. Accurate assessment of citrus canker severity and other diseases is needed for several purposes, including monitoring epidemics and evaluation of germplasm. We compared measurements of citrus canker severity (percent area infected) from automated image analysis to visual estimates by raters and true values using images from five leaf samples (65, 123, 50, 50, and 200 leaves; disease severity from 0 to 60%). Severity on leaves was measured by automated image analysis by (i) basing threshold values on a presample of leaves, or (ii) replacing healthy leaf color on a leaf-by-leaf basis before automating image analysis. Samples 1 to 4 were assessed by three trained plant pathologists, and sample 5 was assessed by an additional 25 raters. Healthy leaf area color replacement gave the most consistent agreement with the true severity data. Using color replacement, agreement with true values based on Lin's concordance correlation coefficient (ρ_c) was 0.93, 0.79, 0.71, 0.85, and 0.89 for each of the samples, respectively. The range and consistency of agreement was generally less good for automated thresholds based on a presample (ρ_c = 0.35-0.90) or visual raters (ρ_c = 0.30-0.94). The constituents of agreement (precision and accuracy) showed similar trends. No one rater or method was best for every leaf sample, but replacing healthy color in each leaf with a standard color before automation of image analysis improved agreement, and was relatively quick (20 s per image). The accuracy and precision of automated image analysis of citrus canker severity can be comparable to unaided, direct visual estimation by many raters.