Enhanced Just Noticeable Difference Model for Images With Pattern Complexity

Semantically Adaptive JND Modeling with Object-Wise Feature Characterization, Context Inhibition and Cross-Object Interaction

Performance bottlenecks in the optimization of JND modeling based on low-level manual visual feature metrics have emerged. High-level semantics bear a considerable impact on perceptual attention and subjective video quality, yet most existing JND models do not adequately account for this impact. This indicates that there is still much room and potential for performance optimization in semantic feature-based JND models. To address this status quo, this paper investigates the response of visual attention induced by heterogeneous semantic features with an eye on three aspects, i.e., object, context, and cross-object, to further improve the efficiency of JND models. On the object side, this paper first focuses on the main semantic features that affect visual attention, including semantic sensitivity, objective area and shape, and central bias. Following that, the coupling role of heterogeneous visual features with HVS perceptual properties are analyzed and quantified. Second, based on the reciprocity of objects and contexts, the contextual complexity is measured to gauge the inhibitory effect of contexts on visual attention. Third, cross-object interactions are dissected using the principle of bias competition, and a semantic attention model is constructed in conjunction with a model of attentional competition. Finally, to build an improved transform domain JND model, a weighting factor is used by fusing the semantic attention model with the basic spatial attention model. Extensive simulation results validate that the proposed JND profile is highly consistent with HVS and highly competitive among state-of-the-art models.

High dynamic range image compression based on the multi-peak S-shaped tone curve

Tone mapping methods aim to compress the high dynamic range (HDR) images so that they can be displayed on common devices. The tone curve plays a key role in many tone mapping methods, which can directly adjust the range of the HDR image. The S-shaped tone curves can produce impressive performances due to their flexibility. However, the conventional S-shaped tone curve in tone mapping methods is single and had the problem of excessive compressing of the dense grayscale areas, resulting in the loss of details in this area, and insufficient compressing of the sparse grayscale areas, resulting in low contrast of tone mapped image. This paper proposes a multi-peak S-shaped (MPS) tone curve to address these problems. Specifically, the grayscale interval of the HDR image is divided according to the significant peak and valley distribution of the grayscale histogram, and each interval is tone mapped by an S-shaped tone curve. We further propose an adaptive S-shaped tone curve based on the luminance adaptation mechanism of the human visual system, which can effectively reduce the compression in the dense grayscale areas and increase the compression in the sparse grayscale areas, preserving details while improving the contrast of tone mapped images. Experiments show that our MPS tone curve replaces the single S-shaped tone curve in relevant methods for better performance and outperforms the state-of-the-art tone mapping methods.

Just Noticeable Difference Model for Images with Color Sensitivity

The just noticeable difference (JND) model reflects the visibility limitations of the human visual system (HVS), which plays an important role in perceptual image/video processing and is commonly applied to perceptual redundancy removal. However, existing JND models are usually constructed by treating the color components of three channels equally, and their estimation of the masking effect is inadequate. In this paper, we introduce visual saliency and color sensitivity modulation to improve the JND model. Firstly, we comprehensively combined contrast masking, pattern masking, and edge protection to estimate the masking effect. Then, the visual saliency of HVS was taken into account to adaptively modulate the masking effect. Finally, we built color sensitivity modulation according to the perceptual sensitivities of HVS, to adjust the sub-JND thresholds of Y, Cb, and Cr components. Thus, the color-sensitivity-based JND model (CSJND) was constructed. Extensive experiments and subjective tests were conducted to verify the effectiveness of the CSJND model. We found that consistency between the CSJND model and HVS was better than existing state-of-the-art JND models.

Pixel-Domain Just Noticeable Difference Modeling with Heterogeneous Color Features

With the rapidly emerging user-generated images, perception compression for color image is an inevitable mission. Whilst in existing just noticeable difference (JND) models, color-oriented features are not fully taken into account for coinciding with HVS perception characteristics, such as sensitivity, attention, and masking. To fully imitate the color perception process, we extract color-related feature parameters as local features, including color edge intensity and color complexity, as well as region-wise features, including color area proportion, color distribution position and color distribution dispersion, and inherent feature irrelevant to color content called color perception difference. Then, the potential interaction among them is analyzed and modeled as color contrast intensity. To utilize them, color uncertainty and color saliency are envisaged to emanate from feature integration in the information communication framework. Finally, color and uncertainty saliency models are applied to improve the conventional JND model, taking the masking and attention effect into consideration. Subjective and objective experiments validate the effectiveness of the proposed model, delivering superior noise concealment capacity compared with start-of-the-art works.

RIHOOP: Robust Invisible Hyperlinks in Offline and Online Photographs

In the era of multimedia and Internet, the quick response (QR) code helps people obtain information from offline to online quickly. However, the QR code is often limited in many scenarios because of its random and dull appearance. Therefore, this article proposes a novel approach to embed hyperlinks into common images, making the hyperlinks invisible for human eyes but detectable for mobile devices equipped with a camera. Our approach is an end-to-end neural network with an encoder to hide messages and a decoder to extract messages. To maintain the hidden message resilient to cameras, we build a distortion network between the encoder and the decoder to augment the encoded images. The distortion network uses differentiable 3-D rendering operations, which can simulate the distortion introduced by camera imaging in both printing and display scenarios. To maintain the visual attraction of the image with hyperlinks, a loss function conforming to the human visual system (HVS) is used to supervise the training of the encoder. Experimental results show that the proposed approach outperforms the previous work on both robustness and quality. Based on the proposed approach, many applications become possible, for example, "image hyperlinks" for advertisement on TV, website, or poster, and "invisible watermark" for copyright protection on digital resources or product packagings.

Toward Top-Down Just Noticeable Difference Estimation of Natural Images

Just noticeable difference (JND) of natural images refers to the maximum pixel intensity change magnitude that typical human visual system (HVS) cannot perceive. Existing efforts on JND estimation mainly dedicate to modeling the diverse masking effects in either/both spatial or/and frequency domains, and then fusing them into an overall JND estimate. In this work, we turn to a dramatically different way to address this problem with a top-down design philosophy. Instead of explicitly formulating and fusing different masking effects in a bottom-up way, the proposed JND estimation model dedicates to first predicting a critical perceptual lossless (CPL) counterpart of the original image and then calculating the difference map between the original image and the predicted CPL image as the JND map. We conduct subjective experiments to determine the critical points of 500 images and find that the distribution of cumulative normalized KLT coefficient energy values over all 500 images at these critical points can be well characterized by a Weibull distribution. Given a testing image, its corresponding critical point is determined by a simple weighted average scheme where the weights are determined by a fitted Weibull distribution function. The performance of the proposed JND model is evaluated explicitly with direct JND prediction and implicitly with two applications including JND-guided noise injection and JND-guided image compression. Experimental results have demonstrated that our proposed JND model can achieve better performance than several latest JND models. In addition, we also compare the proposed JND model with existing visual difference predicator (VDP) metrics in terms of the capability in distortion detection and discrimination. The results indicate that our JND model also has a good performance in this task. The code of this work are available at https://github.com/Zhentao-Liu/KLT-JND.

A Fast Perceptual Surveillance Video Coding (PSVC) Based on Background Model-Driven JND Estimation

Perceptual video coding (PVC) optimization has been an important video coding technique, which can be consistent with the perception characteristics of the human visual system (HVS). Currently, PVC schemes incorporating the just noticeable distortion (JND) model can obtain better performance gain in all PVC schemes. To further accelerate the JND computation for real-time video coding applications (e.g. surveillance video coding and conference video coding), this paper proposes a fast perceptual surveillance video coding (PSVC) scheme based on background model-driven JND estimation method. First, to utilize the surveillance scene characteristics, the computation complexity of JND estimation can be significantly decreased by reusing the content complexity of background regions. Then we apply the perceptive video coding scheme into the background modeling-based surveillance video codec. The proposed scheme adopts background modeling frame as background anchor. Experimental results show that the proposed scheme can yield remarkable time saving of 42.33% maximum and on average 34.76% with approximate bitrate reductions and similar subjective quality, compared to HEVC and other state-of-the-art schemes.

Hierarchical Predictive Coding-Based JND Estimation for Image Compression

The human visual system (HVS) is a hierarchical system, in which visual signals are processed hierarchically. In this paper, the HVS is modeled as a three-level communication system and visual perception is divided into three stages according to the hierarchical predictive coding theory. Then, a novel just noticeable distortion (JND) estimation scheme is proposed. In visual perception, the input signals are predicted constantly and spontaneously in each hierarchy, and neural response is evoked by the central residue and inhibited by surrounding residues. These two types' residues are regarded as the positive and negative visual incentives which cause positive and negative perception effects, respectively. In neuroscience, the effect of incentive on observer is measured by the surprise of this incentive. Thus, we propose a surprise-based measurement method to measure both perception effects. Specifically, considering the biased competition of visual attention, we define the product of the residue self-information (i.e., surprise) and the competition biases as the perceptual surprise to measure the positive perception effect. As for the negative perception effect, it is measured by the average surprise (i.e., the local Shannon entropy). The JND threshold of each stage is estimated individually by considering both perception effects. The total JND threshold is finally obtained by non-linear superposition of three stage thresholds. Furthermore, the proposed JND estimation scheme is incorporated into the codec of Versatile Video Coding for image compression. Experimental results show that the proposed JND model outperforms the relevant existing ones, and over 16% of bit rate can be reduced without jeopardizing the perceptual quality.

Just Noticeable Distortion Profile Inference: A Patch-Level Structural Visibility Learning Approach

In this paper, we propose an effective approach to infer the just noticeable distortion (JND) profile based on patch-level structural visibility learning. Instead of pixel-level JND profile estimation, the image patch, which is regarded as the basic processing unit to better correlate with the human perception, can be further decomposed into three conceptually independent components for visibility estimation. In particular, to incorporate the structural degradation into the patch-level JND model, a deep learning-based structural degradation estimation model is trained to approximate the masking of structural visibility. In order to facilitate the learning process, a JND dataset is further established, including 202 pristine images and 7878 distorted images generated by advanced compression algorithms based on the upcoming Versatile Video Coding (VVC) standard. Extensive experimental results further show the superiority of the proposed approach over the state-of-the-art. Our dataset is available at: https://github.com/ShenXuelin-CityU/PWJNDInfer.

Spatial-Perceptual Embedding with Robust Just Noticeable Difference Model for Color Image Watermarking

In the robust image watermarking framework, watermarks are usually embedded in the direct current (DC) coefficients in discrete cosine transform (DCT) domain, since the DC coefficients have a larger perceptual capacity than any alternating current (AC) coefficients. However, DC coefficients are also excluded from watermark embedding with the consideration of avoiding block artifacts in watermarked images. Studies on human vision suggest that perceptual characteristics can achieve better image fidelity. With this perspective, we propose a novel spatial–perceptual embedding for a color image watermarking algorithm that includes the robust just-noticeable difference (JND) guidance. The logarithmic transform function is used for quantization embedding. Meanwhile, an adaptive quantization step is modeled by incorporating the partial AC coefficients. The novelty and effectiveness of the proposed framework are supported by JND perceptual guidance for spatial pixels. Experiments validate that the proposed watermarking algorithm produces a significantly better performance.

Structure descriptor based on just noticeable difference for texture image classification

Local binary pattern (LBP) and its derivates have been widely used in texture classification. However, LBP-based methods are sensitive to noise, and some structure information represented by non-uniform patterns is lost due to the combination of these patterns. In this paper, a new local structure descriptor based on just noticeable difference (JND) for texture classification is proposed by exploring the spatial and relative intensity correlations among local neighborhood pixels. First, a JND map of the image is computed, and then we attempt to model the correlations among local neighborhood pixels by comparing the absolute differences in intensity between the central pixel and its neighbors with the corresponding JND threshold. A new visual pattern (JNDVP) is designed using modeled correlations to describe image structure. Next, considering that image contrast makes important contributions to structure description, contrast is employed as a weighting factor for JNDVP histogram creation to represent structural and contrast information in a single representation. Finally, the nearest neighborhood classifier is employed for texture classification. Results on two texture image databases demonstrate that the proposed structure descriptor is rotation invariant and more robust to noise than LBP. Moreover, texture classification based on JNDVP outperforms LBP-based methods.

Deep Learning based Picture-Wise Just Noticeable Distortion Prediction Model for Image Compression

Picture Wise Just Noticeable Difference (PW-JND), which accounts for the minimum difference of a picture that human visual system can perceive, can be widely used in perception-oriented image and video processing. However, the conventional Just Noticeable Difference (JND) models calculate the JND threshold for each pixel or sub-band separately, which may not reflect the total masking effect of a picture accurately. In this paper, we propose a deep learning based PW-JND prediction model for image compression. Firstly, we formulate the task of predicting PW-JND as a multi-class classification problem, and propose a framework to transform the multi-class classification problem to a binary classification problem solved by just one binary classifier. Secondly, we construct a deep learning based binary classifier named perceptually lossy/lossless predictor which can predict whether an image is perceptually lossy to another or not. Finally, we propose a sliding window based search strategy to predict PW-JND based on the prediction results of the perceptually lossy/lossless predictor. Experimental results show that the mean accuracy of the perceptually lossy/lossless predictor reaches 92%, and the absolute prediction error of the proposed PW-JND model is 0.79 dB on average, which shows the superiority of the proposed PW-JND model to the conventional JND models.

A Perceptual Distinguishability Predictor For JND-noise-contaminated Images

Just noticeable difference (JND) models are widely used for perceptual redundancy estimation in images and videos. A common method for measuring the accuracy of a JND model is to inject random noise in an image based on the JND model, and check whether the JND-noise-contaminated image is perceptually distinguishable from the original image or not. Also, when comparing the accuracy of two different JND models, the model that produces the JND-noise-contaminated image with better quality at the same level of noise energy is the better model. But in both of these cases, a subjective test is necessary, which is very time consuming and costly. In this paper, we present a full-reference metric called PDP (perceptual distinguishability predictor), which can be used to determine whether a given JND-noise-contaminated image is perceptually distinguishable from the reference image. The proposed metric employs the concept of sparse coding, and extracts a feature vector out of a given image pair. The feature vector is then fed to a multilayer neural network for classification. To train the network, we built a public database of 999 natural images with distinguishbility thresholds for four different JND models obtained from an extensive subjective experiment. The results indicated that PDD achieves high classification accuracy of 97.1%. The proposed method can be used to objectively compare various JND models without performing any subjective test. It can also be used to obtain proper scaling factors to improve the JND thresholds estimated by an arbitrary JND model.