Spatio-temporal frequency separability in area 18 neurons of the cat

Blind Video Quality Prediction by Uncovering Human Video Perceptual Representation

Blind video quality assessment (VQA) has become an increasingly demanding problem in automatically assessing the quality of ever-growing in-the-wild videos. Although efforts have been made to measure temporal distortions, the core to distinguish between VQA and image quality assessment (IQA), the lack of modeling of how the human visual system (HVS) relates to the temporal quality of videos hinders the precise mapping of predicted temporal scores to the human perception. Inspired by the recent discovery of the temporal straightness law of natural videos in the HVS, this paper intends to model the complex temporal distortions of in-the-wild videos in a simple and uniform representation by describing the geometric properties of videos in the visual perceptual domain. A novel videolet, with perceptual representation embedding of a few consecutive frames, is designed as the basic quality measurement unit to quantify temporal distortions by measuring the angular and linear displacements from the straightness law. By combining the predicted score on each videolet, a perceptually temporal quality evaluator (PTQE) is formed to measure the temporal quality of the entire video. Experimental results demonstrate that the perceptual representation in the HVS is an efficient way of predicting subjective temporal quality. Moreover, when combined with spatial quality metrics, PTQE achieves top performance over popular in-the-wild video datasets. More importantly, PTQE requires no additional information beyond the video being assessed, making it applicable to any dataset without parameter tuning. Additionally, the generalizability of PTQE is evaluated on video frame interpolation tasks, demonstrating its potential to benefit temporal-related enhancement tasks.

Reliable, Fast and Stable Contrast Response Function Estimation

A study was conducted to determine stable cortical contrast response functions (CRFs) accurately and repeatedly in the shortest possible experimentation time. The method consisted of searching for experimental temporal aspects (number and duration of trials and number and distribution of contrasts used) with a model based on inhomogeneous Poisson spike trains to varying contrast levels. The set of values providing both short experimental duration and maximizing fit of the CRFs were saved, and then tested on cats' visual cortical neurons. Our analysis revealed that 4 sets of parameters with less or equal to 6 experimental visual contrasts satisfied our premise of obtaining good CRFs' performance in a short recording period, in which the number of trials seems to be the experimental condition that stabilizes the fit.

Exploring and explaining properties of motion processing in biological brains using a neural network

Visual motion perception underpins behaviors ranging from navigation to depth perception and grasping. Our limited access to biological systems constrains our understanding of how motion is processed within the brain. Here we explore properties of motion perception in biological systems by training a neural network to estimate the velocity of image sequences. The network recapitulates key characteristics of motion processing in biological brains, and we use our access to its structure to explore and understand motion (mis)perception. We find that the network captures the biological response to reverse-phi motion in terms of direction. We further find that it overestimates and underestimates the speed of slow and fast reverse-phi motion, respectively, because of the correlation between reverse-phi motion and the spatiotemporal receptive fields tuned to motion in opposite directions. Second, we find that the distribution of spatiotemporal tuning properties in the V1 and middle temporal (MT) layers of the network are similar to those observed in biological systems. We then show that, in comparison to MT units tuned to fast speeds, those tuned to slow speeds primarily receive input from V1 units tuned to high spatial frequency and low temporal frequency. Next, we find that there is a positive correlation between the pattern-motion and speed selectivity of MT units. Finally, we show that the network captures human underestimation of low coherence motion stimuli, and that this is due to pooling of noise and signal motion. These findings provide biologically plausible explanations for well-known phenomena and produce concrete predictions for future psychophysical and neurophysiological experiments.

Screen Content Video Quality Assessment: Subjective and Objective Study

In this paper, we make the first attempt to study the subjective and objective quality assessment for the screen content videos (SCVs). For that, we construct the first large-scale video quality assessment (VQA) database specifically for the SCVs, called the screen content video database (SCVD). This SCVD provides 16 reference SCVs, 800 distorted SCVs, and their corresponding subjective scores, and it is made publicly available for research usage. The distorted SCVs are generated from each reference SCV with 10 distortion types and 5 degradation levels for each distortion type. Each distorted SCV is rated by at least 32 subjects in the subjective test. Furthermore, we propose the first full-reference VQA model for the SCVs, called the spatiotemporal Gabor feature tensor-based model (SGFTM), to objectively evaluate the perceptual quality of the distorted SCVs. This is motivated by the observation that 3D-Gabor filter can well stimulate the visual functions of the human visual system (HVS) on perceiving videos, being more sensitive to the edge and motion information that are often-encountered in the SCVs. Specifically, the proposed SGFTM exploits 3D-Gabor filter to individually extract the spatiotemporal Gabor feature tensors from the reference and distorted SCVs, followed by measuring their similarities and later combining them together through the developed spatiotemporal feature tensor pooling strategy to obtain the final SGFTM score. Experimental results on SCVD have shown that the proposed SGFTM yields a high consistency on the subjective perception of SCV quality and consistently outperforms multiple classical and state-of-the-art image/video quality assessment models.

An Optical Flow-Based Full Reference Video Quality Assessment Algorithm

We present a simple yet effective optical flow-based full-reference video quality assessment (FR-VQA) algorithm for assessing the perceptual quality of natural videos. Our algorithm is based on the premise that local optical flow statistics are affected by distortions and the deviation from pristine flow statistics is proportional to the amount of distortion. We characterize the local flow statistics using the mean, the standard deviation, the coefficient of variation (CV), and the minimum eigenvalue ( λ min ) of the local flow patches. Temporal distortion is estimated as the change in the CV of the distorted flow with respect to the reference flow, and the correlation between λ min of the reference and of the distorted patches. We rely on the robust multi-scale structural similarity index for spatial quality estimation. The computed temporal and spatial distortions, thus, are then pooled using a perceptually motivated heuristic to generate a spatio-temporal quality score. The proposed method is shown to be competitive with the state-of-the-art when evaluated on the LIVE SD database, the EPFL Polimi SD database, and the LIVE Mobile HD database. The distortions considered in these databases include those due to compression, packet-loss, wireless channel errors, and rate-adaptation. Our algorithm is flexible enough to allow for any robust FR spatial distortion metric for spatial distortion estimation. In addition, the proposed method is not only parameter-free but also independent of the choice of the optical flow algorithm. Finally, we show that the replacement of the optical flow vectors in our proposed method with the much coarser block motion vectors also results in an acceptable FR-VQA algorithm. Our algorithm is called the flow similarity index.

Spatiotemporal tuning in mouse primary visual cortex

The neural correlates of visual motion perception have historically been studied in non-human primates. However, the mouse has recently gained popularity as a model for studying vision primarily driven by the hope that the genetic tools available in this species may contribute to our understanding of visual processing in the cortex. A recent calcium-imaging study on the spatiotemporal tuning of mouse striate and extrastriate cortex revealed that neurons in the primary visual cortex (V1) were almost never speed tuned, whereas previous electrophysiological studies in macaques noted around one quarter of V1 neurons appeared to be selective for a particular stimulus speed. We were interested in whether this discrepancy was due to methodological or species differences, so we measured the spatiotemporal tuning of mouse V1 neurons using standard electrophysiological techniques. Using comparable analyses to previous studies of speed tuning, our data showed that speed tuning is rare in mouse V1, which corroborates earlier studies in mouse and points to a species difference in motion processing in early cortex between macaques and other mammals.

Phase sensitivities, excitatory summation fields, and silent suppressive receptive fields of single neurons in the parastriate cortex of the cat

We have recorded single-neuron activity from cytoarchitectonic area 18 of anesthetized (0.4–0.7% isoflurane in 65% N2O-35% O2 gaseous mixture) domestic cats. Neurons were identified as simple or complex on the basis of the ratios between the phase-variant (F1) component and the mean firing rate (F0) of spike responses to optimized (orientation, direction, spatial and temporal frequencies, size) high-contrast, luminance-modulated, sine-wave drifting gratings (simple: F1/F0 spike-response ratios > 1; complex: F1/F0 spike-response ratios < 1). The predominance (∼80%) of simple cells among the neurons recorded from the principal thalamorecipient layers supports the idea that most simple cells in area 18 might constitute a putative early stage in the visual information processing. Apart from the “spike-generating” regions (the classical receptive fields, CRFs), the receptive fields of three-quarters of area 18 neurons contain silent, extraclassical suppressive regions (ECRFs). The spatial extent of summation areas of excitatory responses was negatively correlated with the strength of the ECRF-induced suppression of spike responses. Lowering the stimulus contrast resulted in an expansion of the summation areas of excitatory responses accompanied by a reduction in the strength of the ECRF-induced suppression. The spatial and temporal frequency and orientation tunings of the ECRFs were much broader than those of the CRFs. Hence, the ECRFs of area 18 neurons appear to be largely “inherited” from their dorsal thalamic inputs. In most area 18 cells, costimulation of CRFs and ECRFs resulted in significant increases in F1/F0 spike-response ratios, and thus there was a contextually modulated functional continuum between the simple and complex cells.

Motion tuned spatio-temporal quality assessment of natural videos

There has recently been a great deal of interest in the development of algorithms that objectively measure the integrity of video signals. Since video signals are being delivered to human end users in an increasingly wide array of applications and products, it is important that automatic methods of video quality assessment (VQA) be available that can assist in controlling the quality of video being delivered to this critical audience. Naturally, the quality of motion representation in videos plays an important role in the perception of video quality, yet existing VQA algorithms make little direct use of motion information, thus limiting their effectiveness. We seek to ameliorate this by developing a general, spatio-spectrally localized multiscale framework for evaluating dynamic video fidelity that integrates both spatial and temporal (and spatio-temporal) aspects of distortion assessment. Video quality is evaluated not only in space and time, but also in space-time, by evaluating motion quality along computed motion trajectories. Using this framework, we develop a full reference VQA algorithm for which we coin the term the MOtion-based Video Integrity Evaluation index, or MOVIE index. It is found that the MOVIE index delivers VQA scores that correlate quite closely with human subjective judgment, using the Video Quality Expert Group (VQEG) FRTV Phase 1 database as a test bed. Indeed, the MOVIE index is found to be quite competitive with, and even outperform, algorithms developed and submitted to the VQEG FRTV Phase 1 study, as well as more recent VQA algorithms tested on this database.

Spatial and temporal frequency selectivity of neurons in the middle temporal visual area of new world monkeys (Callithrix jacchus)

Information about the responses of neurons to the spatial and temporal frequencies of visual stimuli is important for understanding the types of computations being performed in different visual areas. We characterized the spatiotemporal selectivity of neurons in the middle temporal area (MT), which is deemed central for the processing of direction and speed of motion. Recordings obtained in marmoset monkeys using high-contrast sine-wave gratings as stimuli revealed that the majority of neurons had bandpass spatial and temporal frequency tuning, and that the selectivity for these parameters was largely separable. Only in about one-third of the cells was inseparable spatiotemporal tuning detected, this typically being in the form of an increase in the optimal temporal frequency as a function of increasing grating spatial frequency. However, most of these interactions were weak, and only 10% of neurons showed spatial frequency-invariant representation of speed. Cells with inseparable spatiotemporal tuning were most commonly found in the infragranular layers, raising the possibility that they form part of the feedback from MT to caudal visual areas. While spatial frequency tuning curves were approximately scale-invariant on a logarithmic scale, temporal frequency tuning curves covering different portions of the spectrum showed marked and systematic changes. Thus, MT neurons can be reasonably described as similarly built spatial frequency filters, each covering a different dynamic range. The small proportion of speed-tuned neurons, together with the laminar position of these units, are compatible with the idea that an explicit neural representation of speed emerges from computations performed in MT.

Spectral receptive field properties of neurons in the feline superior colliculus

The spatio-temporal frequency response profiles of 73 neurons located in the superficial, retino-recipient layers of the feline superior colliculus (SC) were investigated. The majority of the SC cells responded optimally to very low spatial frequencies with a mean of 0.1 cycles/degree (c/deg). The spatial resolution was also low with a mean of 0.31 c/deg. The spatial frequency tuning functions were either low-pass or band-pass with a mean spatial frequency bandwidth of 1.84 octaves. The cells responded optimally to a range of temporal frequencies between 0.74 cycles/s (c/s) and 26.41 c/s with a mean of 6.84 c/s. The majority (68%) of the SC cells showed band-pass temporal frequency tuning with a mean temporal frequency bandwidth of 2.4 octaves, while smaller proportions of the SC units displayed high-pass (19%), low-pass (8%) or broad-band (5%) temporal tuning. Most of the SC units exhibited simple spectral tuning with a single maximum in the spatio-temporal frequency domain, while some neurons were tuned for spatial or temporal frequencies or speed tuned. Further, we found cells excited by gratings moving at high temporal and low spatial frequencies and cells whose activity was suppressed by high velocity movement. The spatio-temporal filter properties of the SC neurons show close similarities to those of their retinal Y and W inputs as well as those of their inputs from the cortical visual motion detector areas, suggesting their common role in motion analysis and related behavioral actions.

Attentional modulation of motion integration of individual neurons in the middle temporal visual area

We examined how spatially directed attention affected the integration of motion in neurons of the middle temporal (MT) area of visual cortex. We recorded from single MT neurons while monkeys performed a motion detection task under two attentional states. Using 0% coherent random dot motion, we estimated the optimal linear transfer function (or kernel) between the global motion and the neuronal response. This linear kernel filtered the random dot motion across direction, speed, and time. Slightly less than one-half of the neurons produced reasonably well defined kernels that also tended to account for both the directional selectivity and responses to coherent motion of different strengths. This subpopulation of cells had faster, more transient, and more robust responses to visual stimuli than neurons with kernels that did not contain well defined regions of integration. For those neurons that had large attentional modulation and produced well defined kernels, we found attention scaled the temporal profile of the transfer function with no appreciable shift in time or change in shape. Thus, for MT neurons described by a linear transfer function, attention produced a multiplicative scaling of the temporal integration window.

Adaptive temporal integration of motion in direction-selective neurons in macaque visual cortex

Direction-selective neurons in the primary visual cortex (V1) and the extrastriate motion area MT/V5 constitute a critical channel that links early cortical mechanisms of spatiotemporal integration to downstream signals that underlie motion perception. We studied how temporal integration in direction-selective cells depends on speed, spatial frequency (SF), and contrast using randomly moving sinusoidal gratings and spike-triggered average (STA) analysis. The window of temporal integration revealed by the STAs varied substantially with stimulus parameters, extending farther back in time for slow motion, high SF, and low contrast. At low speeds and high SF, STA peaks were larger, indicating that a single spike often conveyed more information about the stimulus under conditions in which the mean firing rate was very low. The observed trends were similar in V1 and MT and offer a physiological correlate for a large body of psychophysical data on temporal integration. We applied the same visual stimuli to a model of motion detection based on oriented linear filters (a motion energy model) that incorporated an integrate-and-fire mechanism and found that it did not account for the neuronal data. Our results show that cortical motion processing in V1 and in MT is highly nonlinear and stimulus dependent. They cast considerable doubt on the ability of simple oriented filter models to account for the output of direction-selective neurons in a general manner. Finally, they suggest that spike rate tuning functions may miss important aspects of the neural coding of motion for stimulus conditions that evoke low firing rates.

The neural representation of speed in macaque area MT/V5

Tuning for speed is one key feature of motion-selective neurons in the middle temporal visual area of the macaque cortex (MT, or V5). The present paper asks whether speed is coded in a way that is invariant to the shape of the moving stimulus, and if so, how. When tested with single sine-wave gratings of different spatial and temporal frequencies, MT neurons show a continuum in the degree to which preferred speed depends on spatial frequency. There is some dependence in 75% of MT neurons, and the other 25% maintain speed tuning despite changes in spatial frequency. When tested with stimuli constructed by adding two superimposed sine-wave gratings, the preferred speed of MT neurons becomes less dependent on spatial frequency. Analysis of these responses reveals a speed-tuning nonlinearity that selectively enhances the responses of the neuron when multiple spatial frequencies are present and moving at the same speed. Consistent with the presence of the nonlinearity, MT neurons show speed tuning that is close to form-invariant when the moving stimuli comprise square-wave gratings, which contain multiple spatial frequencies moving at the same speed. We conclude that the neural circuitry in and before MT makes no explicit attempt to render MT neurons speed-tuned for sine-wave gratings, which do not occur in natural scenes. Instead, MT neurons derive form-invariant speed tuning in a way that takes advantage of the multiple spatial frequencies that comprise moving objects in natural scenes.

Effects of attention on orientation-tuning functions of single neurons in macaque cortical area V4

We examined how attention affected the orientation tuning of 262 isolated neurons in extrastriate area V4 and 135 neurons in area V1 of two rhesus monkeys. The animals were trained to perform a delayed match-to-sample task in which oriented stimuli were presented in the receptive field of the neuron being recorded. On some trials the animals were instructed to pay attention to those stimuli, and on other trials they were instructed to pay attention to other stimuli outside the receptive field. In this way, orientation-tuning curves could be constructed from neuronal responses collected in two behavioral states: one in which those stimuli were attended by the animal and one in which those stimuli were ignored by the animal. We fit Gaussians to the neuronal responses to twelve different orientations for each behavioral state. Although attention enhanced the responses of V4 neurons (median 26% increase) and V1 neurons (median 8% increase), selectivity, as measured by the width of its orientation-tuning curve, was not systematically altered by attention. The effects of attention were consistent with a multiplicative scaling of the driven response to all orientations. We also found that attention did not cause systematic changes in the undriven activity of the neurons.

Effect of luminance contrast on the motion aftereffect

The effects of luminance contrast and spatial frequency on the motion aftereffect were investigated. The point of subjective equality for velocity was measured as an index of the motion aftereffect. The largest effect was observed when a low contrast grating (5%) was presented as a test stimulus after adaptation to a high contrast grating (100%) in the low spatial frequency condition (0.8 cycle deg.-1). On the whole, the effect increased with increasing adapting contrast and with decreasing test contrast or spatial frequency. Small effects were observed at high test contrasts. These results were inconsistent with those of Keck, Palella, and Pantle in 1976. Analysis showed that there was no saturation on velocity of the motion aftereffect above 5% of the contrast although Keck, et al. (1976) found that the incremental increases of the effect above 3% adapting contrast were small.