Interobserver agreement: A preliminary investigation into how much is enough?

Behavioral responses of dairy cows and their calves to gradual or abrupt weaning and separation when managed in full- or part-time cow-calf contact systems

There is growing interest in managing cow and calf together for a prolonged period, but concerns remain about how best to wean and separate the cow and calf to minimize stress. One approach is to progressively reduce suckling opportunity over time, as in nature. There is also interest in part-time contact (suckling for part of the day) to improve milk yield for the farmer and potentially reduce stress at weaning and separation. The primary objective of this study was to compare the behavioral responses of dairy cows to gradual or abrupt weaning and separation, when managed either full or part time with their calves; a secondary focus was the vocal responses of calves under these management conditions. In a 3 × 2 factorial design (n = 14/ treatment), dairy cows and their calves (n = 84 in 7 blocks of 12 cow-calf pairs) were assigned to one of 3 dam-contact treatments at birth: (1) full-time contact between the dam and calf, apart from milking times (total 23 h/d) (2) part-time contact between the dam and calf, between morning and afternoon milking only (total 10 h/d); (3) no contact, where the dam and calf were separated after leaving the maternity pen and had no further contact. At wk 8, one of 2 weaning treatments was assigned: (1) gradual weaning by reduced contact time (50%, then 25% of original dam-contact time in wk 8 and 9, respectively), or gradually reduced milk allowance for no-contact calves (50%, then 25% of estimated 12-L milk intake in wk 8 and 9, respectively) until complete milk removal and dam-calf separation at wk 10; (2) abrupt weaning where milk removal occurred simultaneously with dam-calf separation at wk 10, or only milk removal for no-contact calves. Overall, part-time contact did not reduce weaning and separation distress for cows or calves for either weaning method. Part-time cows showed reduced behavioral responses to separation (greater lying time and less searching behavior), especially on the day of and 24 h after separation, but they showed a similarly strong vocal response to separation as full-time cows. Part-time calves made substantially more high-pitched vocalizations than full-time calves at 24 h after separation. Furthermore, gradual weaning by reducing contact time did not seem to better prepare cows or calves for complete milk removal and separation; most behavioral and vocal responses occurred on the day of separation for gradual-weaning cows and calves, but 24 h later, the reverse occurred for abrupt-weaning cows and there was no difference between gradual- and abrupt-weaning calves. Our results suggest that part-time contact and gradual-weaning conditions likely resulted in hunger and expectation for reunion, which together may have exaggerated behavioral responses at separation. Temporary daily separations may not have increased independence of cow and calf, and the gradual-weaning method here may not have reduced milk intake by calves. These elements are important criteria to facilitate the weaning process, so future work should explore ways to gradually reduce milk intake and promote social independence of cow and calf to minimize weaning distress.

Inconsistency between Human Observation and Deep Learning Models: Assessing Validity of Postmortem Computed Tomography Diagnosis of Drowning

Drowning diagnosis is a complicated process in the autopsy, even with the assistance of autopsy imaging and the on-site information from where the body was found. Previous studies have developed well-performed deep learning (DL) models for drowning diagnosis. However, the validity of the DL models was not assessed, raising doubts about whether the learned features accurately represented the medical findings observed by human experts. In this paper, we assessed the medical validity of DL models that had achieved high classification performance for drowning diagnosis. This retrospective study included autopsy cases aged 8-91 years who underwent postmortem computed tomography between 2012 and 2021 (153 drowning and 160 non-drowning cases). We first trained three deep learning models from a previous work and generated saliency maps that highlight important features in the input. To assess the validity of models, pixel-level annotations were created by four radiological technologists and further quantitatively compared with the saliency maps. All the three models demonstrated high classification performance with areas under the receiver operating characteristic curves of 0.94, 0.97, and 0.98, respectively. On the other hand, the assessment results revealed unexpected inconsistency between annotations and models' saliency maps. In fact, each model had, respectively, around 30%, 40%, and 80% of irrelevant areas in the saliency maps, suggesting the predictions of the DL models might be unreliable. The result alerts us in the careful assessment of DL tools, even those with high classification performance.

Comparing Multiple Methods to Measure Procedural Fidelity of Discrete-trial Instruction

Procedural fidelity is the extent to which an intervention is implemented as designed and is an important component of research and practice. There are multiple ways to measure procedural fidelity, and few studies have explored how procedural fidelity varies based on the method of measurement. The current study compared adherence to discrete-trial instruction protocols by behavior technicians with a child with autism when observers used different procedural-fidelity measures. We collected individual-component and individual-trial fidelity with an occurrence-nonoccurrence data sheet and compared these scores to global fidelity and all-or-nothing, 3-point Likert scale, and 5-point Likert scale measurement methods. The all-or-nothing method required all instances of a component or trial be implemented without error to be scored correct. The Likert scales used a rating system to score components and trials. At the component level, we found that the global, 3-point Likert, and 5-point Likert methods were likely to overestimate fidelity and mask component errors, and the all-or-nothing method was unlikely to mask errors. At the trial level, we found that the global and 5-point Likert methods approximated individual-trial fidelity, the 3-point Likert method overestimated fidelity, and the all-or-nothing method underestimated fidelity. The occurrence-nonoccurrence method required the most time to complete, and all-or-nothing by trial required the least. We discuss the implications of measuring procedural fidelity with different methods of measurement, including false positives and false negatives, and provide suggestions for practice and research.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43494-023-00094-w.

Prioritizing choice and assent in the assessment and treatment of food selectivity

Food selectivity affects up to 72% and 45% of individuals with and without disabilities, respectively, and there is a need for interventions that rely on positive, unrestrictive strategies. We evaluated an assessment and treatment package for food selectivity for young children with developmental disabilities that prioritized caregiver collaboration, client autonomy, and did not rely on restrictive procedures (e.g. escape extinction). The process involved: (a) collaborating with caregivers on the selection of foods and design of the children's functional analyses; (b) indirectly and directly measuring food preferences prior to treatment; (c) evaluating the sensitivity of mealtime problem behavior to environmental variables through an interview-informed synthesized contingency analysis (IISCA); and (c) incorporating the assessment results into a progressive treatment process consisting of choice-making opportunities and differential reinforcement of successive approximations to consumption. Children also had the ability to opt in and out of treatment sessions. The treatment was effective in increasing consumption of nonpreferred foods and successfully extended to caregivers. Practical implications and directions for future research are discussed.

Interobserver agreement and procedural fidelity: An odd asymmetry

We examined articles with experiments published in the Journal of Applied Behavior Analysis and in Behavior Analysis in Practice from 2017 through 2021 to determine how frequently procedural fidelity was assessed. When procedural fidelity was assessed, we determined how often a measure of interobserver agreement for those fidelity data was provided. We also determined how often a measure of interobserver agreement for participants' behavior was provided. Across both journals and all years, 54.7% of relevant articles provided a measure of procedural fidelity. Of them, 17.7% provided a measure of interobserver agreement for procedural fidelity. In marked contrast, 96.4% provided interobserver agreement data for participants' behavior. It is unfortunate that applied behavior analysts frequently fail to provide procedural fidelity data and, when they do, often fail to provide interobserver agreement data for the fidelity data. Reviewers for, and editors of, behavior-analytic journals are encouraged to strongly consider the relative value of procedural fidelity and agreement on procedural fidelity measures when rendering recommendations on the suitability of a given submission.

Validity and Reliability Evidence for Assessments Based in Applied Behavior Analysis: A Systematic Review

The current article presents the findings from a systematic review of the available reliability and validity evidence supporting the use of criterion-referenced assessments based on the applied behavior analysis framework. We identified 46 studies that reported reliability and/or validity evidence for six assessments, 37 of which presented reliability evidence and 43 presented validity evidence. Additionally, we extracted and summarized information related to participant characteristics (e.g., age, sex, diagnosis), geographic location, and research setting (e.g., residential facility, home). Overall, we found conflicting support for the use of the assessments. When coupled with the reported usage by behavior analysis professionals, our findings suggest a misalignment between the reportedly used assessments and the number of published studies providing validity and/or reliability evidence. We found inconsistent use of measurement-related vocabulary and that many studies could have been strengthened by conducting different statistical analyses. We provide a summary of studies, findings, and offer recommendations for clinical practice and future measurement research.

The Correlation between Play Behavior, Serum Cortisol and Neutrophil-to-Lymphocyte Ratio in Welfare Assessment of Dairy Calves within the First Month of Life

Welfare assessment of dairy calves is commonly conducted through the observation of their behavior and the analysis of physiological parameters. Despite the large number of studies on this topic, there is a lack of research on the possible correlation between the physiological parameters and behavior at basal level in dairy calves. For this reason, the present study aimed to investigate the possible correlations between serum cortisol, neutrophil-to-lymphocyte ratio (N/L) and locomotor play behavior in 21 Friesian dairy calves within their first month of life. Every week, an arena test after blood collection was performed. A significant positive correlation between serum cortisol levels and N/L (rrm = 0.381; 95% CI = 0.083, 0.617; p = 0.012) and a negative correlation between time spent running and N/L (rrm = −0.351; 95% CI = −0.575, −0.077; p = 0.012) were found. On the contrary, no significant correlation between time spent running and cortisol levels was detected. Cortisol levels and N/L were higher in the newborn and had a constant decrease until reaching a stable value after the 24th day of life. These findings suggest a better predictivity of N/L, compared to serum cortisol, on locomotor play behavior.

On the generality of preference for contingent reinforcement

Reinforcers can be delivered dependent on or independent of responding. Both human and nonhuman animals have shown a preference for contexts involving contingent reinforcement, but the generality of this phenomenon to humans and its implications have not yet been described. We present an integrative summary of studies evaluating preference for contingent versus noncontingent reinforcement, including (a) study participants, reinforcer types, response topographies, and contexts; (b) outcomes; (c) potential variables influencing preference outcomes; and (d) suggestions regarding research to expand behavior analysts' ability to design preferred contexts.