The Effect of Horsfall-Barratt Category Size on the Accuracy and Reliability of Estimates of Pecan Scab Severity

Standard Area Diagrams for Pecan Leaf Scab: Effect of Rater Experience, Location, and Leaf Size on Reliability and Accuracy of Visual Estimates

Assessments of the severity of scab (Venturia effusa), an economically significant disease of pecan, are critical for determining pecan cultivar susceptibility, disease epidemiology, and integrated disease management approaches. We developed a standard area diagram (SAD) set to aid in assessments of pecan leaflet scab. Leaflets with scab lesions were harvested and scanned using a flatbed scanner at 600 dpi, and Fiji (ImageJ) was used to determine the actual percent disease severity. The SADs had 10 leaflets ranging in severity from 0.2 to 48.9%. Forty "small" (1.34 to 7.43 cm2) and 40 "large" (7.67 to 25.9 cm2) leaflet images were randomized for rater assessments. The images were assessed twice by 36 raters, first without and then with the SADs as a guide. Data were subjected to analysis using Lin's concordance correlation coefficient (LCC, pc) to determine the accuracy of ratings and by intraclass correlation coefficient (ICC) analysis to determine interrater reliability. The effects of rater experience, rater location, and leaflet size were also determined. The SADs significantly improved the agreement between raters and the actual values (LCC, pc = 0.70 and 0.84 without and with the SADs, respectively). The reliability of estimates was improved (ICC = 0.54 and 0.82 without and with the SADs, respectively). The effect of rater location on overall concordance was significant without and with the SADs based on an analysis of variance using a generalized linear model and lsmeans separation (P < 0.05). A generalized linear mixed model analysis revealed that there was a significant interaction between rater location, experience, and the use of the SADs, with some raters having greater improvement in generalized bias and concordance. Raters had a significantly better accuracy when rating "small" leaves (LCC, pc = 0.86) compared with "large" leaves (LCC, pc = 0.82) when using the SADs, highlighting the impact of psychophysics on field evaluations of plant disease severity. The proposed SADs will serve as an improved tool for performing pecan leaflet scab assessments by the pecan research community.

Survival Analysis as a Basis for Testing Hypotheses when Using Quantitative Ordinal Scale Disease Severity Data

Disease severity in plant pathology is often measured by the amount of a plant or plant part that exhibits disease symptoms. This is typically assessed using a numerical scale, which allows a standardized, convenient, and quick method of rating. These scales, known as quantitative ordinal scales (QOS), divide the percentage scale into a predetermined number of intervals. There are various ways to analyze these ordinal data, with traditional methods involving the use of midpoint conversion to represent the interval. However, this may not be precise enough, as it is only an estimate of the true value. In this case, the data may be considered interval-censored, meaning that we have some knowledge of the value but not an exact measurement. This type of uncertainty is known as censoring, and techniques that address censoring, such as survival analysis (SA), use all available information and account for this uncertainty. To investigate the pros and cons of using SA with QOS measurements, we conducted a simulation based on three pathosystems. The results showed that SA almost always outperformed midpoint conversion with data analyzed using a t test, particularly when data were not normally distributed. Midpoint conversion is currently a standard procedure. In certain cases, the midpoint approach required a 400% increase in sample size to achieve the same power as the SA method. However, as the mean severity increases, fewer additional samples are needed (approximately an additional 100%), regardless of the assessment method used. Based on these findings, we conclude that SA is a valuable method for enhancing the power of hypothesis testing when analyzing QOS severity data. Future research should investigate the wider use of survival analysis techniques in plant pathology and their potential applications in the discipline.

Understanding the ramifications of quantitative ordinal scales on accuracy of estimates of disease severity and data analysis in plant pathology

The severity of plant diseases, traditionally defined as the proportion of the plant tissue exhibiting symptoms, is a key quantitative variable to know for many diseases but is prone to error. Plant pathologists face many situations in which the measurement by nearest percent estimates (NPEs) of disease severity is time-consuming or impractical. Moreover, rater NPEs of disease severity are notoriously variable. Therefore, NPEs of disease may be of questionable value if severity cannot be determined accurately and reliably. In such situations, researchers have often used a quantitative ordinal scale of measurement-often alleging the time saved, and the ease with which the scale can be learned. Because quantitative ordinal disease scales lack the resolution of the 0 to 100% scale, they are inherently less accurate. We contend that scale design and structure have ramifications for the resulting analysis of data from the ordinal scale data. To minimize inaccuracy and ensure that there is equivalent statistical power when using quantitative ordinal scale data, design of the scales can be optimized for use in the discipline of plant pathology. In this review, we focus on the nature of quantitative ordinal scales used in plant disease assessment. Subsequently, their application and effects will be discussed. Finally, we will review how to optimize quantitative ordinal scales design to allow sufficient accuracy of estimation while maximizing power for hypothesis testing.

Disease Severity Estimates-Effects of Rater Accuracy and Assessment Methods for Comparing Treatments

Assessment of disease severity is required for several purposes in plant pathology; most often, the estimates are made visually. It is established that visual estimates can be inaccurate and unreliable. The ramifications of biased or imprecise estimates by raters have not been fully explored using empirical data, partly because of the logistical difficulties involved in different raters assessing the same leaves for which actual disease has been measured in a replicated experiment with multiple treatments. In this study, nearest percent estimates (NPEs) of Septoria leaf blotch (SLB) on leaves of winter wheat from nontreated and fungicide-treated plots were assessed in both 2006 and 2007 by four raters and compared with assumed actual values measured using image analysis. Lin's concordance correlation (LCC, ρ_c) was used to assess agreement between the two approaches. NPEs were converted to Horsfall-Barratt (HB) midpoints and were compared with actual values. The estimates of SLB severity from fungicide-treated and nontreated plots were analyzed using generalized linear mixed modeling to ascertain effects of rater using both the NPE and HB values. Rater 1 showed good accuracy (ρ_c = 0.986 to 0.999), while raters 3 and 4 were less accurate (ρ_c = 0.205 to 0.936). Conversion to the HB scale had little effect on bias but reduced numerically both precision and accuracy for most raters on most assessment dates (precision, r = -0.001 to -0.132; and accuracy, ρ_c = -0.003 to -0.468). Interrater reliability was also reduced slightly by conversion of estimates to HB midpoint values. Estimates of mean SLB severity were significantly different between image analysis and raters 2, 3, and 4, and there were frequently significant differences among raters (F = 151 to 1,260, P = 0.001 to P < 0.0001). Only on 26 June 2007 did conversion to the HB scale change the means separation ranking of rater estimates. Nonetheless, image analysis and all raters were able to differentiate control and treated-plot treatments (F = 116 to 1,952, P = 0.002 to P < 0.0001, depending on date and rater). Conversion of NPEs to the HB scale tended to reduce F values slightly (2006: NPEs, F = 116 to 276, P = 0.002 to 0.0005; and, for the HB-converted values, F = 101 to 270, P = 0.002 to 0.0005; 2007: NPEs, F = 164 to 1,952, P = 0.001 to P < 0.0001; and, for HB-converted values, F = 126 to 1,633, P = 0.002 to P < 0.0001). The results reaffirm the need for accurate and reliable disease assessment to minimize over- or underestimates compared with actual disease, and the data we present support the view that, where multiple raters are deployed, they should be assigned in a manner to reduce any potential effect of rater differences on the analysis.

What interval characteristics make a good categorical disease assessment scale?

Plant pathologists most often obtain quantitative information on disease severity using visual assessments. Category scales have been used for assessing plant disease severity in field experiments, epidemiological studies, and for screening germplasm. The most widely used category scale is the Horsfall-Barratt (H-B) scale, but reports show that estimates of disease severity using the H-B scale are less precise compared with nearest percent estimates (NPEs) using the 0 to 100% ratio scale. Few studies have compared different category scales. The objective of this study was to compare NPEs, the H-B midpoint converted data, and four different linear category scales (5 and 10% increments, with and without additional grades at low severity [0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 15.0, 20.0…100%, and 0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 20.0, 30.0…100%, respectively]). Results of simulations based on known distributions of disease estimation using the type II error rate (the risk of failing to reject H0 when H0 is false) showed that at disease severity ≤ 5%, a 10% category scale had a greater probability of failing to reject H0 when H0 is false compared with all other methods, while the H-B scale performed least well at 20 to 50% severity. The 5% category scale performed as well as NPEs except when disease severity was ≤ 1%. Both the 5 and 10% category scales with the additional grades included performed as well as NPEs. These results were confirmed with a mixed model analysis and bootstrap analysis of the original rater assessment data. A better knowledge of the advantages and disadvantages of category scale types will provide a basis for plant pathologists and plant breeders seeking to maximize accuracy and reliability of assessments to make an informed decision when choosing a disease assessment method.