Cue combination in a combined feature contrast detection and figure identification task

Science, evidence, law, and justice

For nearly 25 y, the Committee on Science, Technology, and Law (CSTL), of the National Academies of Sciences, Engineering, and Medicine, has brought together distinguished members of the science and law communities to stimulate discussions that would lead to a better understanding of the role of science in legal decisions and government policies and to a better understanding of the legal and regulatory frameworks that govern the conduct of science. Under the leadership of recent CSTL co-chairs David Baltimore and David Tatel, and CSTL director Anne-Marie Mazza, the committee has overseen many interdisciplinary discussions and workshops, such as the international summits on human genome editing and the science of implicit bias, and has delivered advisory consensus reports focusing on topics of broad societal importance, such as dual use research in the life sciences, voting systems, and advances in neural science research using organoids and chimeras. One of the most influential CSTL activities concerns the use of forensic evidence by law enforcement and the courts, with emphasis on the scientific validity of forensic methods and the role of forensic testimony in bringing about justice. As coeditors of this Special Feature, CSTL alumni Tom Albright and Jennifer Mnookin have recruited articles at the intersection of science and law that reveal an emerging scientific revolution of forensic practice, which we hope will engage a broad community of scientists, legal scholars, and members of the public with interest in science-based legal policy and justice reform.

Synergy of spatial frequency and orientation bandwidth in texture segregation

Defining target textures by increased bandwidths in spatial frequency and orientation, we observed strong cue combination effects in a combined texture figure detection and discrimination task. Performance for double-cue targets was better than predicted by independent processing of either cue and even better than predicted from linear cue integration. Application of a texture-processing model revealed that the oversummative cue combination effect is captured by calculating a low-level summary statistic (\(\Delta CE_m\)), which describes the differential contrast energy to target and reference textures, from multiple scales and orientations, and integrating this statistic across channels with a winner-take-all rule. Modeling detection performance using a signal detection theory framework showed that the observers' sensitivity to single-cue and double-cue texture targets, measured in \(d^{\prime }\) units, could be reproduced with plausible settings for filter and noise parameters. These results challenge models assuming separate channeling of elementary features and their later integration, since oversummative cue combination effects appear as an inherent property of local energy mechanisms, at least for spatial frequency and orientation bandwidth-modulated textures.

Object Localization Does Not Imply Awareness of Object Category at the Break of Continuous Flash Suppression

In continuous flash suppression (CFS), a dynamic noise masker, presented to one eye, suppresses conscious perception of a test stimulus, presented to the other eye, until the suppressed stimulus comes to awareness after few seconds. But what do we see breaking the dominance of the masker in the transition period? We addressed this question with a dual-task in which observers indicated (i) whether the test object was left or right of the fixation mark (localization) and (ii) whether it was a face or a house (categorization). As done recently Stein et al. (2011a), we used two experimental varieties to rule out confounds with decisional strategy. In the terminated mode, stimulus and masker were presented for distinct durations, and the observers were asked to give both judgments at the end of the trial. In the self-paced mode, presentation lasted until the observers responded. In the self-paced mode, b-CFS durations for object categorization were about half a second longer than for object localization. In the terminated mode, correct categorization rates were consistently lower than correct detection rates, measured at five duration intervals ranging up to 2 s. In both experiments we observed an upright face advantage compared to inverted faces and houses, as concurrently reported in b-CFS studies. Our findings reveal that more time is necessary to enable observers judging the nature of the object, compared to judging that there is “something other” than the noise which can be localized, but not recognized. This suggests gradual transitions in the first break of CFS. Further, the results imply that suppression is such that no cues to object identity are conveyed in potential “leaks” of CFS (Gelbard-Sagiv et al., 2016).

Effects of Spatial Frequency Similarity and Dissimilarity on Contour Integration

We examined the effects of spatial frequency similarity and dissimilarity on human contour integration under various conditions of uncertainty. Participants performed a temporal 2AFC contour detection task. Spatial frequency jitter up to 3.0 octaves was applied either to background elements, or to contour and background elements, or to none of both. Results converge on four major findings. (1) Contours defined by spatial frequency similarity alone are only scarcely visible, suggesting the absence of specialized cortical routines for shape detection based on spatial frequency similarity. (2) When orientation collinearity and spatial frequency similarity are combined along a contour, performance amplifies far beyond probability summation when compared to the fully heterogenous condition but only to a margin compatible with probability summation when compared to the fully homogenous case. (3) Psychometric functions are steeper but not shifted for homogenous contours in heterogenous backgrounds indicating an advantageous signal-to-noise ratio. The additional similarity cue therefore not so much improves contour detection performance but primarily reduces observer uncertainty about whether a potential candidate is a contour or just a false positive. (4) Contour integration is a broadband mechanism which is only moderately impaired by spatial frequency dissimilarity.

Integration trumps selection in object recognition

Finding and recognizing objects is a fundamental task of vision. Objects can be defined by several "cues" (color, luminance, texture, etc.), and humans can integrate sensory cues to improve detection and recognition [1-3]. Cortical mechanisms fuse information from multiple cues [4], and shape-selective neural mechanisms can display cue invariance by responding to a given shape independent of the visual cue defining it [5-8]. Selective attention, in contrast, improves recognition by isolating a subset of the visual information [9]. Humans can select single features (red or vertical) within a perceptual dimension (color or orientation), giving faster and more accurate responses to items having the attended feature [10, 11]. Attention elevates neural responses and sharpens neural tuning to the attended feature, as shown by studies in psychophysics and modeling [11, 12], imaging [13-16], and single-cell and neural population recordings [17, 18]. Besides single features, attention can select whole objects [19-21]. Objects are among the suggested "units" of attention because attention to a single feature of an object causes the selection of all of its features [19-21]. Here, we pit integration against attentional selection in object recognition. We find, first, that humans can integrate information near optimally from several perceptual dimensions (color, texture, luminance) to improve recognition. They cannot, however, isolate a single dimension even when the other dimensions provide task-irrelevant, potentially conflicting information. For object recognition, it appears that there is mandatory integration of information from multiple dimensions of visual experience. The advantage afforded by this integration, however, comes at the expense of attentional selection.

Component processes in contour integration: a direct comparison between snakes and ladders in a detection and a shape discrimination task

In contour integration, a relevant question is whether snakes and ladders are processed similarly. Higher presentation time thresholds for ladders in detection tasks indicate this is not the case. However, in a detection task only processing differences at the level of element linking and possibly contour localization might be picked up, while differences at the shape encoding level cannot be noticed. In this study, we make a direct comparison of detection and shape discrimination tasks to investigate if processing differences in the visual system between snakes and ladders are limited to contour detection or extend to higher level contour processing, like shape encoding. Stimuli consisted of elements that were oriented collinearly (snakes) or orthogonally (ladders) to the contour path and were surrounded by randomly oriented background elements. In two tasks, six experienced subjects either detected the contour when presented with a contour and a completely random stimulus or performed a shape discrimination task when presented with two contours with different curvature. Presentation time was varied in 9 steps between 8 and 492 ms. By applying a generalized linear mixed model we found that differences in snake and ladder processing are not limited to a detection stage but are also apparent at a shape encoding stage.

The role of binocular disparity in rapid scene and pattern recognition

We investigated the contribution of binocular disparity to the rapid recognition of scenes and simpler spatial patterns using a paradigm combining backward masked stimulus presentation and short-term match-to-sample recognition. First, we showed that binocular disparity did not contribute significantly to the recognition of briefly presented natural and artificial scenes, even when the availability of monocular cues was reduced. Subsequently, using dense random dot stereograms as stimuli, we showed that observers were in principle able to extract spatial patterns defined only by disparity under brief, masked presentations. Comparing our results with the predictions from a cue-summation model, we showed that combining disparity with luminance did not per se disrupt the processing of disparity. Our results suggest that the rapid recognition of scenes is mediated mostly by a monocular comparison of the images, although we can rely on stereo in fast pattern recognition.

Neural representation of feature synergy

Interactive non-linear cooperation of different feature dimensions, feature synergy, has been studied in psychophysics, but the neural mechanism is unknown. The present study investigated the neural representation of feature synergy of two second-order visual features by combining electroencephalography (EEG) with the signal detection theory (SDT). Two kinds of a 27-by-27 array of Gabor patches were presented in a random order; a reference stimulus which has no segregated region, and a target stimulus whose inner region differed in spatial frequency, orientation, or both from the surround. Subjects performed a Yes-No discrimination of whether the inner region was different from the surround, while EEG signals were recorded from 62 locations. When the SDT measure showed feature synergy, EEG activity showed a long-lasting enhancement starting at 130 ms around the inferior temporal region. In contrast, no EEG modulation was observed when feature synergy was not present. Thus, our combined approach demonstrates that non-linear cooperation between different features is represented by neural activity starting at 130 ms post-stimulus in the ventral visual stream.

Visual detection and identification are not the same: evidence from psychophysics and fMRI

Sometimes object detection as opposed to identification is sufficient to initiate the appropriate action. To explore the neural origin of behavioural differences between the two tasks, we combine psychophysical measurements and fMRI, specifically contrasting shape detection versus identification of a figure. This figure consisted of Gabor elements being oriented differently from those in the background. We equalized performance levels for detection and identification by adjusting orientation differences accordingly for each observer. Hence, stimulus saliency was constant for both tasks allowing a differentiation between the activations specific for detection versus identification processes. Identification yielded higher psychophysical thresholds, slower reaction times and increased hemodynamic activations in the lateral-occipital complex (LOC) and an adjacent area in the collateral sulcus (CoS). Additional analysis using cortex-based alignment revealed four voxel-clusters differentially activated by the tasks, situated in the inferior parietal lobe, the precuneus, the anterior cingulum and the medial frontal gyrus. Our results indicate partly separated cortical mechanisms for object detection and identification.

The influence of orientation jitter and motion on contour saliency and object identification

One of the ultimate goals of vision research is to understand how some elements are grouped together and differentiated from others to form object representations in a complex visual scene. There exists an extensive literature on this grouping/segmentation problem, but most of the studies have used un-recognizable stimuli that have little to do with object recognition per se. We used Gabor-rendered outlines of real-world objects to study some relationships between bottom-up and top-down processes in both spatial- and motion form perception. We manipulated low-level properties, such as element orientation and local motion, while incorporating higher-level properties, such as object complexity and identity, and found that adding local motion improved overall performance in both object detection and object identification tasks. Adding orientation jitter effectively decreased object detection performance in both static and motion conditions, and increased reaction time for identification in the static condition. Orientation jitter had much less effect on reaction times for identification in the local motion condition than in the static condition. Both contour properties ("good continuation") and object properties (identifiability) had a positive effect on detection and reaction time for identification.

Summation of perceptual cues in natural visual scenes

Natural visual scenes are rich in information, and any neural system analysing them must piece together the many messages from large arrays of diverse feature detectors. It is known how threshold detection of compound visual stimuli (sinusoidal gratings) is determined by their components' thresholds. We investigate whether similar combination rules apply to the perception of the complex and suprathreshold visual elements in naturalistic visual images. Observers gave magnitude estimations (ratings) of the perceived differences between pairs of images made from photographs of natural scenes. Images in some pairs differed along one stimulus dimension such as object colour, location, size or blur. But, for other image pairs, there were composite differences along two dimensions (e.g. both colour and object-location might change). We examined whether the ratings for such composite pairs could be predicted from the two ratings for the respective pairs in which only one stimulus dimension had changed. We found a pooling relationship similar to that proposed for simple stimuli: Minkowski summation with exponent 2.84 yielded the best predictive power (r=0.96), an exponent similar to that generally reported for compound grating detection. This suggests that theories based on detecting simple stimuli can encompass visual processing of complex, suprathreshold stimuli.

Efficient integration across spatial frequencies for letter identification in foveal and peripheral vision

Objects in natural scenes are spatially broadband; in contrast, feature detectors in the early stages of visual processing are narrowly tuned in spatial frequency. Earlier studies of feature integration using gratings suggested that integration across spatial frequencies is suboptimal. Here we re-examined this conclusion using a letter identification task at the fovea and at 10 deg in the lower visual field. We found that integration across narrow-band (1-octave) spatial frequency components of letter stimuli is optimal in the fovea. Surprisingly, this optimality is preserved in the periphery, even though feature integration is known to be deficient in the periphery from studies of other form-vision tasks such as crowding. A model that is otherwise a white-noise ideal observer except for a limited spatial resolution defined by the human contrast sensitivity function and using internal templates slightly wider in bandwidth than the stimuli is able to account for the human data. Our findings suggest that deficiency in feature integration found in peripheral vision is not across spatial frequencies.

The electrophysiological correlate of contour integration is similar for color and luminance mechanisms

Contour integration perceptually links together similarly oriented line elements hidden between randomly oriented distracters. To investigate how contour integration depends on early sensory processing, we compared the electrophysiological correlate of contour integration of elements defined by luminance (black-and-white) or isoluminant color (red-and-green) contrasts. Detection performance for color- and luminance-defined contours (both open and closed) was matched. Detectable contours elicited a negative shift over posterior electrodes starting 220 ms after stimulus onset. The shift occurred for both color and luminance contrasts, even when possible luminance artifacts in red-and-green stimuli were masked. This indicates a common physiological processing stream for orientation-based contour integration of red-and-green and black-and-white elements.