Non-linear global pooling in the discrimination of circular and non-circular shapes

Mask-Pyramid Network: A Novel Panoptic Segmentation Method

In this paper, we introduce a novel panoptic segmentation method called the Mask-Pyramid Network. Existing Mask RCNN-based methods first generate a large number of box proposals and then filter them at each feature level, which requires a lot of computational resources, while most of the box proposals are suppressed and discarded in the Non-Maximum Suppression process. Additionally, for panoptic segmentation, it is a problem to properly fuse the semantic segmentation results with the Mask RCNN-produced instance segmentation results. To address these issues, we propose a new mask pyramid mechanism to distinguish objects and generate much fewer proposals by referring to existing segmented masks, so as to reduce computing resource consumption. The Mask-Pyramid Network generates object proposals and predicts masks from larger to smaller sizes. It records the pixel area occupied by the larger object masks, and then only generates proposals on the unoccupied areas. Each object mask is represented as a H × W × 1 logit, which fits well in format with the semantic segmentation logits. By applying SoftMax to the concatenated semantic and instance segmentation logits, it is easy and natural to fuse both segmentation results. We empirically demonstrate that the proposed Mask-Pyramid Network achieves comparable accuracy performance on the Cityscapes and COCO datasets. Furthermore, we demonstrate the computational efficiency of the proposed method and obtain competitive results.

Shape discrimination in peripheral vision: Addressing pragmatic limitations of M-scaling radial frequency patterns

Peripheral worsening in shape discrimination (SD) can be compensated by size-scaling of peripheral stimuli. However, such scaling results in production of large stimuli that occupy a vast range of eccentricities. We used six proportionally decreasing spatial scales to address this pragmatic limitation and to explore how shape discrimination varies with radius in the nasal visual field. Five participants with normal vision discriminated circles and radial frequency (RF) patterns presented nasally to the fixation point at 5°, 10°, 15° and 20°. Stimuli were scaled with the nasal cortical magnification factor (nCMF) from a central stimulus in six spatial scales, which varied from 0.125 to 1, where 1 corresponded to 1.2° radius. Thresholds expressed in Weber fractions remained constant at eccentricities up to 20° regardless of the spatial scale. Weber fractions for the smaller spatial scales (0.125-0.5) were higher and more variable than for the larger spatial scales (0.75-1), yet still constant across periphery. The results provide evidence that peripheral shape discrimination is constrained by low-level properties, such as eccentricity, and can be predicted by the cortical magnification theory. However, above the peripheral modulation resolution limits, RF shape discrimination is based on the proportion between the modulation amplitude and the radius for larger scales (0.75-1), and demonstrates peripheral scale invariance for these stimuli. For eccentric shape discrimination tests, stimuli with low spatial frequency, high contrast, and radii corresponding to SS 0.75-0.875 should be used to ensure constant Weber fractions, small variability, and peripheral stimuli that are not excessively magnified.

Gaze patterns during presentation of fixed and random phase radial frequency patterns

Radial frequency (RF) patterns, circles which have had their radius modulated as a function of their polar angle, have been used in the examination of the integration of contour information around closed contour patterns. Typically, these patterns have been presented in a random orientation from trial-to-trial in order to maintain spatial uncertainty as to the location of the deformation on the pattern, as it may affect observer strategy and performance. However, the effect of fixed and random orientation (phase) on observer gaze strategies used to discriminate RF patterns has not been directly tested. This study compared fixation patterns across four conditions: fixed phase single cycle; random phase single cycle; fixed phase three cycle; and random phase three cycle RF3 patterns. The results showed that observers fixated on the known location of deformation for the fixed phase single cycle condition but used a more central fixation for the other three conditions. This strategy had a significant effect on observer thresholds for the fixed phase single cycle condition, with greater adherence to the strategy resulting in lower thresholds. It was also found that for the single cycle patterns observers tended to fixate on different locations on the pattern: on the maximum orientation difference from circular for the fixed phase pattern; and on the point of maximum curvature for the random phase pattern. These differences in gaze patterns are likely driven by the underlying local or global processing of the fixed or random phase single cycle patterns, respectively.

Impact of microsaccades on visual shape processing

Sensitivity to subtle changes in the shape of visual objects has been attributed to the existence of global pooling mechanisms that integrate local form information across space. Although global pooling is typically demonstrated under steady fixation, other work suggests prolonged fixation can lead to a collapse of global structure. Here, we ask whether small ballistic eye movements that naturally occur during periods of fixation affect the global processing of radial frequency (RF) patterns-closed contours created by sinusoidally modulating the radius of a circle. Observers were asked to discriminate the shapes of circular patterns and RF-modulated patterns while fixational eye movements were recorded binocularly at 500 Hz. Microsaccades were detected using a velocity-based algorithm, allowing trials to be sorted according to the relative timing of stimulus and microsaccade onset. Results revealed clear perisaccadic changes in shape discrimination thresholds. Performance was impaired when microsaccades occurred close to stimulus onset, but facilitated when they occurred shortly afterward. In contrast, global integration of shape was unaffected by the timing of microsaccades. These findings suggest that microsaccades alter the discrimination sensitivity to briefly presented shapes but do not disrupt the spatial pooling of local form signals.NEW & NOTEWORTHY Microsaccades cause rapid displacement of visual images during fixation and dramatically alter the perception of basic image features. However, their effect on more complex aspects of visual processing is not well understood. Here, we demonstrate a dissociation in the impact of microsaccades on shape perception. Although overall shape discrimination performance is modulated around the time of microsaccades, the pooling efficiency of global mechanisms that combine local form information across space remains unaffected.

A Multitask-Aided Transfer Learning-Based Diagnostic Framework for Bearings under Inconsistent Working Conditions

Rolling element bearings are a vital part of rotating machines and their sudden failure can result in huge economic losses as well as physical causalities. Popular bearing fault diagnosis techniques include statistical feature analysis of time, frequency, or time-frequency domain data. These engineered features are susceptible to variations under inconsistent machine operation due to the non-stationary, non-linear, and complex nature of the recorded vibration signals. To address these issues, numerous deep learning-based frameworks have been proposed in the literature. However, the logical reasoning behind crack severities and the longer training times needed to identify multiple health characteristics at the same time still pose challenges. Therefore, in this work, a diagnosis framework is proposed that uses higher-order spectral analysis and multitask learning (MTL), while also incorporating transfer learning (TL). The idea is to first preprocess the vibration signals recorded from a bearing to look for distinct patterns for a given fault type under inconsistent working conditions, e.g., variable motor speeds and loads, multiple crack severities, compound faults, and ample noise. Later, these bispectra are provided as an input to the proposed MTL-based convolutional neural network (CNN) to identify the speed and the health conditions, simultaneously. Finally, the TL-based approach is adopted to identify bearing faults in the presence of multiple crack severities. The proposed diagnostic framework is evaluated on several datasets and the experimental results are compared with several state-of-the-art diagnostic techniques to validate the superiority of the proposed model under inconsistent working conditions.

Eccentricity Effect of Deformation Detection for Radial Frequency Patterns With Their Centers at Fixation Point

We measured the eccentricity effect of deformation thresholds of circular contours for two types of the radial frequency (RF) patterns with their centers at the fixation point: constant circular contour frequency (CCF) RF patterns and constant RF magnified (retino-cortical scaling) RF patterns. We varied the eccentricity by changing the mean radius of the RF patterns while keeping the centers of the RF patterns at the fixation point. Our peripheral stimulus presentation was distinguished from previous studies which have simply translated RF patterns at different locations in the visual field. Sensitivity for such shape discrimination fell off as the moderate and high CCF patterns were presented on more eccentric sites but did not as the low CCF patterns. However, sensitivity held constant as the magnified RF patterns were presented on more eccentric sites, indicating that the eccentricity effects observed for the high and moderate CCF patterns were neutralized by retinocortical mapping. Notably, sensitivity for the magnified RF patterns with large radii (4°–16°) presented in the peripheral field revealed a similar RF dependence observed for RF patterns with small radii (0.25°–1.0°) presented at the fovea in previous studies.

Sensitivity to curvature deformations along closed contours

Human observers are exquisitely sensitive to curvature deformations along a circular closed contour (Wilkinson, Wilson, & Habak, 1998; Hess, Wang, & Dakin, 1999; Loffler, Wilson, & Wilkinson, 2003). Such remarkable sensitivity has been attributed to the curvature encoding scheme used by V4 neurons, which typically are assumed to be equally sensitive to curvature at all polar angles (Pasupathy & Connor, 2001, 2002; Carlson, Rasquinha, Zhang, & Connor, 2011). To test the assumption that detection thresholds for curvature deformations are invariant across polar angles, we used a novel stimulus class we call Difference of Gaussian (DoG) contours that allowed us to independently manipulate the amplitude, angular frequency, and polar angle of curvature of a closed-contour shape while measuring contour-curvature thresholds. Our results demonstrate that (a) detection thresholds were higher when observers were uncertain about the location of the curvature deformation, but on average, thresholds did not vary significantly across 24 polar angles; (b) the direction and magnitude of the oblique effect varies across individuals; (c) there is a strong association between detecting a contour deformation and identifying its location; (d) curvature detectors may serve as labeled lines.

The processing of compound radial frequency patterns

Radial frequency (RF) patterns can be combined to construct complex shapes. Previous studies have suggested that such complex shapes may be encoded by multiple, narrowly-tuned RF shape channels. To test this hypothesis, thresholds were measured for detection and discrimination of various combinations of two RF components. Results show evidence of summation: sensitivity for the compounds was better than that for the components, with little effect of the components' relative phase. If both RF components are processed separately at the point of detection, they would combine by probability summation (PS), resulting in only a small increase in sensitivity for the compound compared to the components. Summation exceeding the prediction of PS suggests a form of additive summation (AS) by a common mechanism. Data were compared to predictions of winner-take-all, where only the strongest component contributes to detection, a single channel AS model, and multi-channel PS and AS models. The multi-channel PS and AS models were modelled under both Fixed and Matched Attention Window scenarios, the former assuming a single internal noise source for both components and compounds or different internal noise sources for components and compounds respectively. The winner-take-all and single channel models could be rejected. Of the remaining models, the best performing one was an AS model with a Fixed Attention Window, consistent with detection being mediated by channels that are efficiently combined and limited by a single source of noise for both components and compounds.

Evaluating spatiotemporal interactions between shapes

Spatiotemporal interactions between stimuli can alter the perceived curvature along the outline of a shape (Habak, Wilkinson, Zakher, & Wilson, 2004; Habak, Wilkinson, & Wilson, 2006). To better understand these interactions, we used a forward and backward masking paradigm with radial frequency (RF) contours while measuring RF detection thresholds. In Experiment 1, we presented a mask alongside a target contour and altered the stimulus onset asynchrony between this target-mask pair and a temporal mask. We found that a temporal mask increased thresholds when it preceded the target-mask stimulus by 130-180 ms but decreased thresholds when it followed the target-stimulus mask by 180 ms. Furthermore, Experiment 2 demonstrated that the effects of temporal and spatial masks are approximately additive. We discuss these findings in relation to theories of transient and sustained channels in vision.

Scotopic contour and shape discrimination using radial frequency patterns

Radial frequency (RF) patterns are valuable tools for investigations of contour integration and shape discrimination. Under photopic conditions, healthy observers can detect deformations from circularity in RF patterns as small as 3 seconds of arc. Such fine discrimination may be facilitated by cortical curvature detectors or global shape-detecting mechanisms that favor a closed contour. Rods make up 95% of photoreceptors in the retina, but we know very little about how spatial information is processed by rod-mediated pathways. We measured scotopic radial deformation discrimination using both full and partly occluded RF pattern stimuli. We found radial deformation thresholds of around 2–3 minutes of arc for stimuli with a wide range of radii and RFs. When parts of the stimulus were occluded, scotopic thresholds improved up to the point that three or four cycles of modulation were visible; no further improvement occurred with the addition of more visible cycles. When only one to three cycles were visible, an increase in curvature per cycle became important, allowing observers to detect smaller deformations from circularity. Our results indicate that the scotopic radial deformation thresholds for the stimuli tested are not dependent on global circularity cues but are instead mediated by local curvature cues.

Radial frequency patterns describe a small and perceptually distinct subset of all possible planar shapes

The visual system is exposed to a vast number of shapes and objects. Yet, human object recognition is effortless, fast and largely independent of naturally occurring transformations such as position and scale. The precise mechanisms of shape encoding are still largely unknown. Radial frequency (RF) patterns are a special class of closed contours defined by modulation of a circle's radius. These patterns have been frequently and successfully used as stimuli in vision science to investigate aspects of shape processing. Given their mathematical properties, RF patterns can not represent any arbitrary shape, but the ability to generate more complex, biologically relevant, shapes depicting the outlines of objects such as fruits or human heads raises the possibility that RF patterns span a representative subset of possible shapes. However, this assumption has not been tested before. Here we show that only a small fraction of all possible shapes can be represented by RF patterns and that this small fraction is perceptually distinct from the general class of all possible shapes. Specifically, we derive a general measure for the distance of a given shape's outline from the set of RF patterns, allowing us to scan large numbers of object outlines automatically. We find that only between 1% and 6% of naturally smooth outlines can be exactly represented by RF patterns. We present results from a visual search experiment, which revealed that searching an RF pattern among non-radial frequency patterns is efficient, whereas searching an RF pattern among other RF patterns is inefficient (and vice versa). These results suggest that RF patterns represent only a restricted subset of possible planar shapes and that results obtained with this special class of stimuli can not simply be expected to generalise to any arbitrary planar shape.

Phase-selective masking with radial frequency contours

Sensitivity to changes in the shape of a closed-contour figure is affected by surrounding figures (Vision Research 44 (2004) 2815-2823). We examined how between-contour masking depends on radial frequency. Experiment 1 replicated previous studies that found that masking between adjacent radial frequency (RF) patterns was greatest when the two shapes were phase aligned, and that the magnitude of masking declined approximately linearly with increasing phase offsets. In addition, we found that the effect of phase offset on masking was very similar for RFs ranging from 3 to 8, a result that suggests that sensitivity to phase decreases with increasing radial frequency. Experiment 2 tested this idea and found that phase discrimination threshold for single cycles of curvature was approximately proportional to radial frequency. Experiment 3 showed that both curvature maxima and minima contribute to phase dependent masking between RF contours. Together, Experiments 1-3 demonstrate that the strength of phase-dependent masking does not depend on RF, but is related to sensitivity for phase shifts in isolated contours, and is affected by both positive and negative curvature extrema. We discuss these results in relation to properties of curvature sensitive neurons.

Rejecting probability summation for radial frequency patterns, not so Quick!

Radial frequency (RF) patterns are used to assess how the visual system processes shape. They are thought to be detected globally. This is supported by studies that have found summation for RF patterns to be greater than what is possible if the parts were being independently detected and performance only then improved with an increasing number of cycles by probability summation between them. However, the model of probability summation employed in these previous studies was based on High Threshold Theory (HTT), rather than Signal Detection Theory (SDT). We conducted rating scale experiments to investigate the receiver operating characteristics. We find these are of the curved form predicted by SDT, rather than the straight lines predicted by HTT. This means that to test probability summation we must use a model based on SDT. We conducted a set of summation experiments finding that thresholds decrease as the number of modulated cycles increases at approximately the same rate as previously found. As this could be consistent with either additive or probability summation, we performed maximum-likelihood fitting of a set of summation models (Matlab code provided in our Supplementary material) and assessed the fits using cross validation. We find we are not able to distinguish whether the responses to the parts of an RF pattern are combined by additive or probability summation, because the predictions are too similar. We present similar results for summation between separate RF patterns, suggesting that the summation process there may be the same as that within a single RF.

Imagining circles--Empirical data and a perceptual model for the arc-size illusion

An essential part of visual object recognition is the evaluation of the curvature of both an object's outline as well as the contours on its surface. We studied a striking illusion of visual curvature--the arc-size illusion (ASI)--to gain insight into the visual coding of curvature. In the ASI, short arcs are perceived as flatter (less curved) compared to longer arcs of the same radius. We investigated if and how the ASI depends on (i) the physical size of the stimulus and (ii) on the length of the arc. Our results show that perceived curvature monotonically increases with arc length up to an arc angle of about 60°, thereafter remaining constant and equal to the perceived curvature of a full circle. We investigated if the misjudgment of curvature in the ASI translates into predictable biases for three other perceptual tasks: (i) judging the position of the centre of circular arcs; (ii) judging if two circular arcs fall on the circumference of the same (invisible) circle and (iii) interpolating the position of a point on the circumference of a circle defined by two circular arcs. We found that the biases in all the above tasks were reliably predicted by the same bias mediating the ASI. We present a simple model, based on the central angle subtended by an arc, that captures the data for all tasks. Importantly, we argue that the ASI and related biases are a consequence of the fact that an object's curvature is perceived as constant with viewing distance, in other words is perceptually scale invariant.

Behavioral evidence for a functional link between low- and mid-level visual perception in the autism spectrum

BACKGROUND

Most investigations of visuo-perceptual abilities in the Autism Spectrum (AS) are level-specific, using tasks that selectively solicit either lower- (i.e., spatial frequency sensitivity), mid- (i.e., pattern discrimination) or higher-level processes (i.e., face identification) along the visual hierarchy. Less is known about how alterations at one level of processing (i.e., low-level) interact with that of another (i.e., mid-level). The aim of this study was to assess whether manipulating the physical properties (luminance vs texture) of local contour elements of a mid-level, visual pattern interferes with the discrimination of that pattern in a differential manner for individuals with AS.

METHODS

Twenty-nine AS individuals and thirty control participants (range 14-27 years) were asked to discriminate between perfect circles and Radial Frequency Patterns (RFP) of two, three, five, and 10 radial frequencies (RF), or deformations along the pattern's contour. When RFP have few deformations (<five RF), a global, pattern analysis is needed for shape discrimination. Conversely, when RFP contain many deformations (≥10 RF), discrimination is dependent on the analysis of local deformations along the RFP contour. The effect of manipulating RF on RFP discrimination was assessed for RFP whose local contour elements were defined by either luminance or texture information, the latter previously found less efficiently processed in AS individuals.

RESULTS

Two separate mixed factorial ANOVAs [2 (Group)×4 (RF)] were conducted on mean deformation thresholds for luminance- and texture-defined conditions. A significant Group×RF interaction was found for the luminance-defined condition where thresholds were higher in the AS group for the two and three RF conditions; no between-group differences were found for the five and 10 RF conditions. A significant main effect of group was identified for the texture-defined condition, where mean thresholds were higher for the AS group across all RF conditions assessed (two, three, five and 10); a Group×RF interaction effect was not found. Performance for each RFP condition was not affected across group by either chronological age or intelligence, as measured by either Weschler scales or Raven Progressive Matrices.

CONCLUSIONS

The ability of AS individuals to discriminate a circular pattern is differentially affected by the availability (number of deformations along the RFP contour) and type (luminance vs texture) of local, low-level elements defining its contour. Performance is unaffected in AS when RFP discrimination is dependent on the analysis of local deformations of luminance-defined contour elements, but decreased across all RF conditions when local contour elements are texture-defined. These results suggest that efficient pattern perception in AS is functionally related to the efficacy with which its local elements are processed, indicative of an early origin for altered mid-level, pattern perception in AS.

Visual search targeting either local or global perceptual processes differs as a function of autistic-like traits in the typically developing population

Relative to low scorers, high scorers on the autism-spectrum quotient (AQ) show enhanced performance on the embedded figures test and the radial frequency search task (RFST), which has been attributed to both enhanced local processing and differences in combining global percepts. We investigate the role of local and global processing further using the RFST in four experiments. High AQ adults maintained a consistent advantage in search speed across diverse target-distracter stimulus conditions. This advantage may reflect enhanced local processing of curvature in early stages of the form vision pathway and superior global detection of shape primitives. However, more probable is the presence of a superior search process that enables a consistent search advantage at both levels of processing.

Near their thresholds for detection, shapes are discriminated by the angular separation of their corners

Observers make sense of scenes by parsing images on the retina into meaningful objects. This ability is retained for line drawings, demonstrating that critical information is concentrated at object boundaries. Information theoretic studies argue for further concentration at points of maximum curvature, or corners, on such boundaries [1]–[3] suggesting that the relative positions of such corners might be important in defining shape. In this study we use patterns subtly deformed from circular, by a sinusoidal modulation of radius, in order to measure threshold sensitivity to shape change. By examining the ability of observers to discriminate between patterns of different frequency and/or number of cycles of modulation in a 2x2 forced choice task we were able to show, psychophysically, that difference in a single cue, the periodicity of the corners (specifically the polar angle between two points of maximum curvature) was sufficient to allow discrimination of two patterns near their thresholds for detection. We conclude that patterns could be considered as labelled for this measure. These results suggest that a small number of such labels might be sufficient to identify an object.

Set-size effects for sampled shapes: experiments and model

The location of imperfections or heterogeneities in shapes and contours often correlates with points of interest in a visual scene. Investigating the detection of such heterogeneities provides clues as to the mechanisms processing simple shapes and contours. We determined set-size effects (e.g., sensitivity to single target detection as distractor number increases) for sampled contours to investigate how the visual system combines information across space. Stimuli were shapes sampled by oriented Gabor patches: circles and high-amplitude RF4 and RF8 radial frequency patterns with Gabor orientations tangential to the shape. Subjects had to detect a deviation in orientation of one element ("heterogeneity"). Heterogeneity detection sensitivity was measured for a range (7-40) of equally spaced (2.3-0.4°) elements. In a second condition, performance was measured when elements sampled a part of the shapes. We either varied partial contour length for a fixed (7) set-size, co-varying inter-element spacing, or set-size for a fixed spacing (0.7°), co-varying partial contour length. Surprisingly, set-size effects (poorer performance with more elements) are rarely seen. Set-size effects only occur for shapes containing concavities (RF4 and RF8) and when spacing is fixed. When elements are regularly spaced, detection performance improves with set-size for all shapes. When set-size is fixed and spacing varied, performance improves with decreasing spacing. Thus, when an increase in set-size and a decrease in spacing co-occur, the effect of spacing dominates, suggesting that inter-element spacing, not set-size, is the critical parameter for sampled shapes. We propose a model for the processing of simple shapes based on V4 curvature units with late noise, incorporating spacing, average shape curvature, and the number of segments with constant sign of curvature contained in the shape, which accurately accounts for our experimental results, making testable predictions for a variety of simple shapes.

Detecting shapes in noise: tuning characteristics of global shape mechanisms

The proportion of signal elements embedded in noise needed to detect a signal is a standard tool for investigating motion perception. This paradigm was applied to the shape domain to determine how local information is pooled into a global percept. Stimulus arrays consisted of oriented Gabor elements that sampled the circumference of concentric radial frequency (RF) patterns. Individual Gabors were oriented tangentially to the shape (signal) or randomly (noise). In different conditions, signal elements were located randomly within the entire array or constrained to fall along one of the concentric contours. Coherence thresholds were measured for RF patterns with various frequencies (number of corners) and amplitudes (“sharpness” of corners). Coherence thresholds (about 10% = 15 elements) were lowest for circular shapes. Manipulating shape frequency or amplitude showed a range where thresholds remain unaffected (frequency ≤ RF4; amplitude ≤ 0.05). Increasing either parameter caused thresholds to rise. Compared to circles, thresholds increased by approximately four times for RF13 and five times for amplitudes of 0.3. Confining the signals to individual contours significantly reduced the number of elements needed to reach threshold (between 4 and 6), independent of the total number of elements on the contour or contour shape. Finally, adding external noise to the orientation of the elements had a greater effect on detection thresholds than adding noise to their position. These results provide evidence for a series of highly sensitive, shape-specific analysers which sum information globally but only from within specific annuli. These global mechanisms are tuned to position and orientation of local elements from which they pool information. The overall performance for arrays of elements can be explained by the sensitivity of multiple, independent concentric shape detectors rather than a single detector integrating information widely across space (e.g. Glass pattern detector).

Detecting global form: separate processes required for Glass and radial frequency patterns

Global processing of form information has been studied extensively using both Glass and radial frequency (RF) patterns. Models, with common early stages, have been proposed for the detection of properties of both pattern types but human performance has not been examined to determine whether the two pattern types interact in the manner this would suggest. The experiments here investigated whether low RF patterns and concentric Glass patterns, which are thought to tap the same level of processing in form-vision, are detected by a common mechanism. Six observers participated in two series of masking experiments. First: sensitivity to the presence of either coherent structure, or contour deformation, was assessed. The computational model predicted that detection of one pattern would be masked by the other. Second: a further experiment examined position coding. The model predicted that localizing the center of form in a Glass pattern would be affected by the presence of an RF pattern: sensitivity to a change of location should be reduced and the apparent location should be drawn toward the center of the masking pattern. However, the results observed in all experiments were inconsistent with the interaction predicted by the models, suggesting that separate neural mechanisms for global processing of signal are required to process these two patterns, and also indicating that the models need to be altered to preclude the interactions that were predicted but not obtained.