1
|
Pan Q, Liu Q, Huang W. NID-DETR: A novel model for accurate target detection in dark environments. Sci Rep 2025; 15:16002. [PMID: 40341210 PMCID: PMC12062268 DOI: 10.1038/s41598-025-98173-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/30/2024] [Accepted: 04/09/2025] [Indexed: 05/10/2025] Open
Abstract
Target detection in low-light conditions poses significant challenges due to reduced contrast, increased noise, and color distortion, all of which adversely affect detection accuracy and robustness. Effective low-light target detection is crucial for reliable vision in critical applications such as surveillance, autonomous driving, and underwater exploration. Current mainstream algorithms face challenges in extracting meaningful features under low-light conditions, which significantly limits their effectiveness. Furthermore, existing vision Transformer models demonstrate high computational complexity, indicating a need for further optimization and enhancement. Initially, we enhance the dataset during model training to optimize machine vision perception. Subsequently, we design an inverted residual cascade structure module to effectively address the inefficiencies in the global attention window mechanism. Finally, in the target detection output layer, we adopt strategies to reduce concatenation operations and optimize small object detection heads to decrease the model parameter count and improve precision. The dataset is divided into training, testing, and validation sets in a 7:2:1 ratio. Validation on the low-light dataset demonstrates a reduction of 27% in model parameters, with improvements of 2.4%, 4.8%, and 2% in AP50:95, AP50, and AP75, respectively. Our model outperforms both the best baseline and other state-of-the-art models. These experimental results underscore the effectiveness of our proposed approach.
Collapse
Affiliation(s)
- Qingyuan Pan
- School of Computer Science and Engineering, Wuhan institute of Technology, Wuhan, 430205, China
- Hubei Provincial Key Laboratory of Intelligent Robots, Wuhan, 430205, China
| | - Qiang Liu
- School of Computer Science and Engineering, Wuhan institute of Technology, Wuhan, 430205, China
- Hubei Provincial Key Laboratory of Intelligent Robots, Wuhan, 430205, China
| | - Wei Huang
- School of Computer Science and Engineering, Wuhan institute of Technology, Wuhan, 430205, China.
- Wuhan I-Boron Photoelectric Technology Co., Ltd, Wuhan, 430205, China.
| |
Collapse
|
2
|
He M, Wang R, Zhang M, Lv F, Wang Y, Zhou F, Bian X. SwinLightGAN a study of low-light image enhancement algorithms using depth residuals and transformer techniques. Sci Rep 2025; 15:12151. [PMID: 40204793 PMCID: PMC11982214 DOI: 10.1038/s41598-025-95329-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2024] [Accepted: 03/20/2025] [Indexed: 04/11/2025] Open
Abstract
Contemporary algorithms for enhancing images in low-light conditions prioritize improving brightness and contrast but often neglect improving image details. This study introduces the Swin Transformer-based Light-enhancing Generative Adversarial Network (SwinLightGAN), a novel generative adversarial network (GAN) that effectively enhances image details under low-light conditions. The network integrates a generator model based on a Residual Jumping U-shaped Network (U-Net) architecture for precise local detail extraction with an illumination network enhanced by Shifted Window Transformer (Swin Transformer) technology that captures multi-scale spatial features and global contexts. This combination produces high-quality images that resemble those taken in normal lighting conditions, retaining intricate details. Through adversarial training that employs discriminators operating at multiple scales and a blend of loss functions, SwinLightGAN ensures a seamless distinction between generated and authentic images, ensuring superior enhancement quality. Extensive experimental analysis on multiple unpaired datasets demonstrates SwinLightGAN's outstanding performance. The system achieves Naturalness Image Quality Evaluator (NIQE) scores ranging from 5.193 to 5.397, Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) scores from 28.879 to 32.040, and Patch-based Image Quality Evaluator (PIQE) scores from 38.280 to 44.479, highlighting its efficacy in delivering high-quality enhancements across diverse metrics.
Collapse
Affiliation(s)
- Min He
- School of Information Engineering, Yancheng Institute of Technology, Yancheng, 224051, China
| | - Rugang Wang
- School of Information Engineering, Yancheng Institute of Technology, Yancheng, 224051, China.
| | - Mingyang Zhang
- School of Information Engineering, Yancheng Institute of Technology, Yancheng, 224051, China
| | - Feiyang Lv
- School of Information Engineering, Yancheng Institute of Technology, Yancheng, 224051, China
| | - Yuanyuan Wang
- School of Information Engineering, Yancheng Institute of Technology, Yancheng, 224051, China
| | - Feng Zhou
- School of Information Engineering, Yancheng Institute of Technology, Yancheng, 224051, China
| | - Xuesheng Bian
- School of Information Engineering, Yancheng Institute of Technology, Yancheng, 224051, China
| |
Collapse
|
3
|
Liu W, Pang J, Zhang B, Wang J, Liu B, Tao D. See Degraded Objects: A Physics-Guided Approach for Object Detection in Adverse Environments. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2025; 34:2198-2212. [PMID: 40138227 DOI: 10.1109/tip.2025.3551533] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/29/2025]
Abstract
In adverse environments, the detector often fails to detect degraded objects because they are almost invisible and their features are weakened by the environment. Common approaches involve image enhancement to support detection, but they inevitably introduce human-invisible noise that negatively impacts the detector. In this work, we propose a physics-guided approach for object detection in adverse environments, which gives a straightforward solution that injects the physical priors into the detector, enabling it to detect poorly visible objects. The physical priors, derived from the imaging mechanism and image property, include environment prior and frequency prior. The environment prior is generated from the physical model, e.g., the atmospheric model, which reflects the density of environmental noise. The frequency prior is explored based on an observation that the amplitude spectrum could highlight object regions from the background. The proposed two priors are complementary in principle. Furthermore, we present a physics-guided loss that incorporates a novel weight item, which is estimated by applying the membership function on physical priors and could capture the extent of degradation. By backpropagating the physics-guided loss, physics knowledge is injected into the detector to aid in locating degraded objects. We conduct experiments in synthetic foggy environment, real foggy environment, and real underwater scenario. The results demonstrate that our method is effective and achieves state-of-the-art performance. The code is available at https://github.com/PangJian123/See-Degraded-Objects.
Collapse
|
4
|
Wu K, Huang J, Ma Y, Fan F, Ma J. Mutually Reinforcing Learning of Decoupled Degradation and Diffusion Enhancement for Unpaired Low-Light Image Lightening. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2025; 34:2020-2035. [PMID: 40146646 DOI: 10.1109/tip.2025.3553070] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/29/2025]
Abstract
Denoising Diffusion Probabilistic Model (DDPM) has demonstrated exceptional performance in low-light enhancement task. However, the dependency on paired training datas has left the generality of DDPM in low-light enhancement largely untapped. Therefore, this paper proposes a mutually reinforcing learning framework of decoupled degradation and diffusion enhancement, named MRLIE, which leverages style guidance from unpaired low-light images to generate pseudo-image pairs that are consistent with the target domain, thereby optimizing the latter diffusion enhancement network in a supervised manner. During the degradation process, the diffusion loss of fixed enhancement network serves as a evaluation metric for structure consistency and is combined with adversarial style loss to form the optimization objective for degradation network. Such loss design ensures that scene structure information is retained during the degradation process. During the enhancement process, the degradation network with frozen parameters continuously generates pseudo-paired low-/normal-light image pairs as training datas, thus the diffusion enhancement network could be progressively optimized. On the whole, the two processes are interdependent and could achieve cooperative improvement in terms of degradation realism and enhancement quality through iterative optimization. Additionally, we propose the Retinex-based decoupled degradation strategy for simulating the complex degradation in real low-light imaging, which ensures the color correction and noise suppression capabilities of latter diffusion enhancement network. Extensive experiments show that MRLIE can achieve promising results and better generality across various datasets.
Collapse
|
5
|
Wu W, Weng J, Zhang P, Wang X, Yang W, Jiang J. Interpretable Optimization-Inspired Unfolding Network for Low-Light Image Enhancement. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2025; 47:2545-2562. [PMID: 40030787 DOI: 10.1109/tpami.2024.3524538] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/05/2025]
Abstract
Retinex model-based methods have shown to be effective in layer-wise manipulation with well-designed priors for low-light image enhancement (LLIE). However, the hand-crafted priors and conventional optimization algorithm adopted to solve the layer decomposition problem result in the lack of adaptivity and efficiency. To this end, this paper proposes a Retinex-based deep unfolding network (URetinex-Net++), which unfolds an optimization problem into a learnable network to decompose a low-light image into reflectance and illumination layers. By formulating the decomposition problem as an implicit priors regularized model, three learning-based modules are carefully designed, responsible for data-dependent initialization, high-efficient unfolding optimization, and fairly-flexible component adjustment, respectively. Particularly, the proposed unfolding optimization module, introducing two networks to adaptively fit implicit priors in the data-driven manner, can realize noise suppression and details preservation for decomposed components. URetinex-Net++ is a further augmented version of URetinex-Net, which introduces a cross-stage fusion block to alleviate the color defect in URetinex-Net. Therefore, boosted performance on LLIE can be obtained in both visual quality and quantitative metrics, where only a few parameters are introduced and little time is cost. Extensive experiments on real-world low-light images qualitatively and quantitatively demonstrate the effectiveness and superiority of the proposed URetinex-Net++ over state-of-the-art methods.
Collapse
|
6
|
Xie C, Fei L, Tao H, Hu Y, Zhou W, Hoe JT, Hu W, Tan YP. Residual Quotient Learning for Zero-Reference Low-Light Image Enhancement. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2024; PP:365-378. [PMID: 40030647 DOI: 10.1109/tip.2024.3519997] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/05/2025]
Abstract
Recently, neural networks have become the dominant approach to low-light image enhancement (LLIE), with at least one-third of them adopting a Retinex-related architecture. However, through in-depth analysis, we contend that this most widely accepted LLIE structure is suboptimal, particularly when addressing the non-uniform illumination commonly observed in natural images. In this paper, we present a novel variant learning framework, termed residual quotient learning, to substantially alleviate this issue. Instead of following the existing Retinex-related decomposition-enhancement-reconstruction process, our basic idea is to explicitly reformulate the light enhancement task as adaptively predicting the latent quotient with reference to the original low-light input using a residual learning fashion. By leveraging the proposed residual quotient learning, we develop a lightweight yet effective network called ResQ-Net. This network features enhanced non-uniform illumination modeling capabilities, making it more suitable for real-world LLIE tasks. Moreover, due to its well-designed structure and reference-free loss function, ResQ-Net is flexible in training as it allows for zero-reference optimization, which further enhances the generalization and adaptability of our entire framework. Extensive experiments on various benchmark datasets demonstrate the merits and effectiveness of the proposed residual quotient learning, and our trained ResQ-Net outperforms state-of-the-art methods both qualitatively and quantitatively. Furthermore, a practical application in dark face detection is explored, and the preliminary results confirm the potential and feasibility of our method in real-world scenarios.
Collapse
|
7
|
Wang W, Luo R, Yang W, Liu J. Unsupervised Illumination Adaptation for Low-Light Vision. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024; 46:5951-5966. [PMID: 38536689 DOI: 10.1109/tpami.2024.3382108] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/07/2024]
Abstract
Insufficient lighting poses challenges to both human and machine visual analytics. While existing low-light enhancement methods prioritize human visual perception, they often neglect machine vision and high-level semantics. In this paper, we make pioneering efforts to build an illumination enhancement model for high-level vision. Drawing inspiration from camera response functions, our model could enhance images from the machine vision perspective despite being lightweight in architecture and simple in formulation. We also introduce two approaches that leverage knowledge from base enhancement curves and self-supervised pretext tasks to train for different downstream normal-to-low-light adaptation scenarios. Our proposed framework overcomes the limitations of existing algorithms without requiring access to labeled data in low-light conditions. It facilitates more effective illumination restoration and feature alignment, significantly improving the performance of downstream tasks in a plug-and-play manner. This research advances the field of low-light machine analytics and broadly applies to various high-level vision tasks, including classification, face detection, optical flow estimation, and video action recognition.
Collapse
|
8
|
Wang X, Huang L, Li M, Han C, Liu X, Nie T. Fast, Zero-Reference Low-Light Image Enhancement with Camera Response Model. SENSORS (BASEL, SWITZERLAND) 2024; 24:5019. [PMID: 39124066 PMCID: PMC11314879 DOI: 10.3390/s24155019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/05/2024] [Revised: 07/26/2024] [Accepted: 08/01/2024] [Indexed: 08/12/2024]
Abstract
Low-light images are prevalent in intelligent monitoring and many other applications, with low brightness hindering further processing. Although low-light image enhancement can reduce the influence of such problems, current methods often involve a complex network structure or many iterations, which are not conducive to their efficiency. This paper proposes a Zero-Reference Camera Response Network using a camera response model to achieve efficient enhancement for arbitrary low-light images. A double-layer parameter-generating network with a streamlined structure is established to extract the exposure ratio K from the radiation map, which is obtained by inverting the input through a camera response function. Then, K is used as the parameter of a brightness transformation function for one transformation on the low-light image to realize enhancement. In addition, a contrast-preserving brightness loss and an edge-preserving smoothness loss are designed without the requirement for references from the dataset. Both can further retain some key information in the inputs to improve precision. The enhancement is simplified and can reach more than twice the speed of similar methods. Extensive experiments on several LLIE datasets and the DARK FACE face detection dataset fully demonstrate our method's advantages, both subjectively and objectively.
Collapse
Affiliation(s)
- Xiaofeng Wang
- Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China; (X.W.)
- University of Chinese Academy of Sciences, Beijing 100049, China
| | - Liang Huang
- Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China; (X.W.)
| | - Mingxuan Li
- Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China; (X.W.)
| | - Chengshan Han
- Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China; (X.W.)
| | - Xin Liu
- Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China; (X.W.)
| | - Ting Nie
- Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China; (X.W.)
| |
Collapse
|
9
|
Zhang H, Xiao L, Cao X, Foroosh H. Multiple Adverse Weather Conditions Adaptation for Object Detection via Causal Intervention. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024; 46:1742-1756. [PMID: 35412971 DOI: 10.1109/tpami.2022.3166765] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Most state-of-the-art object detection methods have achieved impressive perfomrace on several public benchmarks, which are trained with high definition images. However, existing detectors are often sensitive to the visual variations and out-of-distribution data due to the domain gap caused by various confounders, e.g. the adverse weathre conditions. To bridge the gap, previous methods have been mainly exploring domain alignment, which requires to collect an amount of domain-specific training samples. In this paper, we introduce a novel domain adaptation model to discover a weather condition invariant feature representation. Specifically, we first employ a memory network to develop a confounder dictionary, which stores prototypes of object features under various scenarios. To guarantee the representativeness of each prototype in the dictionary, a dynamic item extraction strategy is used to update the memory dictionary. After that, we introduce a causal intervention reasoning module to explore the invariant representation of a specific object under different weather conditions. Finally, a categorical consistency regularization is used to constrain the similarities between categories in order to automatically search for the aligned instances among distinct domains. Experiments are conducted on several public benchmarks (RTTS, Foggy-Cityscapes, RID, and BDD 100K) with state-of-the-art performance achieved under multiple weather conditions.
Collapse
|
10
|
Tao J, Wu H, Ni Z, Jin Z, Zhong C. MTIE-Net: Multi-technology fusion of low-light image enhancement network. PLoS One 2024; 19:e0297984. [PMID: 38306351 PMCID: PMC10836710 DOI: 10.1371/journal.pone.0297984] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2023] [Accepted: 01/14/2024] [Indexed: 02/04/2024] Open
Abstract
Images obtained in low-light scenes are often accompanied by problems such as low visibility, blurred details, and color distortion, enhancing them can effectively improve the visual effect and provide favorable conditions for advanced visual tasks. In this study, we propose a Multi-Technology Fusion of Low-light Image Enhancement Network (MTIE-Net) that modularizes the enhancement task. MTIE-Net consists of a residual dense decomposition network (RDD-Net) based on Retinex theory, an encoder-decoder denoising network (EDD-Net), and a parallel mixed attention-based self-calibrated illumination enhancement network (PCE-Net). The low-light image is first decomposed by RDD-Net into a lighting map and reflectance map; EDD-Net is used to process noise in the reflectance map; Finally, the lighting map is fused with the denoised reflectance map as an input to PCE-Net, using the Fourier transform for illumination enhancement and detail recovery in the frequency domain. Numerous experimental results show that MTIE-Net outperforms the comparison methods in terms of image visual quality enhancement improvement, denoising, and detail recovery. The application in nighttime face detection also fully demonstrates its promise as a pre-processing means in practical applications.
Collapse
Affiliation(s)
- Jing Tao
- Automation and Information School of Automation and Information Engineering, Sichuan University of Science & Engineering, Zigong, Sichuan Province, China
- Artificial Intelligence Key Laboratory of Sichuan Province, Zigong, Sichuan Province, China
| | - Hao Wu
- Automation and Information School of Automation and Information Engineering, Sichuan University of Science & Engineering, Zigong, Sichuan Province, China
- Artificial Intelligence Key Laboratory of Sichuan Province, Zigong, Sichuan Province, China
| | - Zhihao Ni
- Automation and Information School of Automation and Information Engineering, Sichuan University of Science & Engineering, Zigong, Sichuan Province, China
- Artificial Intelligence Key Laboratory of Sichuan Province, Zigong, Sichuan Province, China
| | - Zhongyang Jin
- Automation and Information School of Automation and Information Engineering, Sichuan University of Science & Engineering, Zigong, Sichuan Province, China
- Artificial Intelligence Key Laboratory of Sichuan Province, Zigong, Sichuan Province, China
| | - Changhua Zhong
- Automation and Information School of Automation and Information Engineering, Sichuan University of Science & Engineering, Zigong, Sichuan Province, China
- Artificial Intelligence Key Laboratory of Sichuan Province, Zigong, Sichuan Province, China
| |
Collapse
|
11
|
Diaz-Ramirez VH, Juarez-Salazar R, Gonzalez-Ruiz M, Adeyemi VA. Restoration of Binocular Images Degraded by Optical Scattering through Estimation of Atmospheric Coefficients. SENSORS (BASEL, SWITZERLAND) 2023; 23:8918. [PMID: 37960616 PMCID: PMC10649635 DOI: 10.3390/s23218918] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/09/2023] [Revised: 10/27/2023] [Accepted: 10/31/2023] [Indexed: 11/15/2023]
Abstract
A binocular vision-based approach for the restoration of images captured in a scattering medium is presented. The scene depth is computed by triangulation using stereo matching. Next, the atmospheric parameters of the medium are determined with an introduced estimator based on the Monte Carlo method. Finally, image restoration is performed using an atmospheric optics model. The proposed approach effectively suppresses optical scattering effects without introducing noticeable artifacts in processed images. The accuracy of the proposed approach in the estimation of atmospheric parameters and image restoration is evaluated using synthetic hazy images constructed from a well-known database. The practical viability of our approach is also confirmed through a real experiment for depth estimation, atmospheric parameter estimation, and image restoration in a scattering medium. The results highlight the applicability of our approach in computer vision applications in challenging atmospheric conditions.
Collapse
Affiliation(s)
- Victor H. Diaz-Ramirez
- Instituto Politécnico Nacional—CITEDI, Ave. Instituto Politécnico Nacional 1310, Tijuana 22310, Mexico; (M.G.-R.); (V.A.A.)
| | - Rigoberto Juarez-Salazar
- CONAHCYT, Instituto Politécnico Nacional—CITEDI, Ave. Instituto Politécnico Nacional 1310, Tijuana 22310, Mexico;
| | - Martin Gonzalez-Ruiz
- Instituto Politécnico Nacional—CITEDI, Ave. Instituto Politécnico Nacional 1310, Tijuana 22310, Mexico; (M.G.-R.); (V.A.A.)
| | - Vincent Ademola Adeyemi
- Instituto Politécnico Nacional—CITEDI, Ave. Instituto Politécnico Nacional 1310, Tijuana 22310, Mexico; (M.G.-R.); (V.A.A.)
| |
Collapse
|
12
|
Liang X, Chen X, Ren K, Miao X, Chen Z, Jin Y. Low-light image enhancement via adaptive frequency decomposition network. Sci Rep 2023; 13:14107. [PMID: 37644042 PMCID: PMC10465598 DOI: 10.1038/s41598-023-40899-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Accepted: 08/17/2023] [Indexed: 08/31/2023] Open
Abstract
Images captured in low light conditions suffer from low visibility, blurred details and strong noise, resulting in unpleasant visual appearance and poor performance of high level visual tasks. To address these problems, existing approaches have attempted to enhance the visibility of low-light images using convolutional neural networks (CNN). However, due to the insufficient consideration of the characteristics of the information of different frequency layers in the image, most of them yield blurry details and amplified noise. In this work, to fully extract and utilize these information, we proposed a novel Adaptive Frequency Decomposition Network (AFDNet) for low-light image enhancement. An Adaptive Frequency Decomposition (AFD) module is designed to adaptively extract low and high frequency information of different granularities. Specifically, the low-frequency information is employed for contrast enhancement and noise suppression in low-scale space and high-frequency information is for detail restoration in high-scale space. Meanwhile, a new frequency loss function are proposed to guarantee AFDNet's recovery capability for different frequency information. Extensive experiments on various publicly available datasets show that AFDNet outperforms the existing state-of-the-art methods both quantitatively and visually. In addition, our results showed that the performance of the face detection can be effectively improved by using AFDNet as pre-processing.
Collapse
Affiliation(s)
- Xiwen Liang
- School of Electronic Information and Automation, Tianjin University of Science and Technology, Tianjin, 300222, China
| | - Xiaoyan Chen
- School of Electronic Information and Automation, Tianjin University of Science and Technology, Tianjin, 300222, China.
| | - Keying Ren
- School of Electronic Information and Automation, Tianjin University of Science and Technology, Tianjin, 300222, China
| | - Xia Miao
- School of Electronic Information and Automation, Tianjin University of Science and Technology, Tianjin, 300222, China
| | - Zhihui Chen
- School of Electronic Information and Automation, Tianjin University of Science and Technology, Tianjin, 300222, China
| | - Yutao Jin
- School of Electronic Information and Automation, Tianjin University of Science and Technology, Tianjin, 300222, China
| |
Collapse
|
13
|
Chao K, Song W, Shao S, Liu D, Liu X, Zhao X. CUI-Net: a correcting uneven illumination net for low-light image enhancement. Sci Rep 2023; 13:12894. [PMID: 37558723 PMCID: PMC10412593 DOI: 10.1038/s41598-023-39524-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Accepted: 07/26/2023] [Indexed: 08/11/2023] Open
Abstract
Uneven lighting conditions often occur during real-life photography, such as images taken at night that may have both low-light dark areas and high-light overexposed areas. Traditional algorithms for enhancing low-light areas also increase the brightness of overexposed areas, affecting the overall visual effect of the image. Therefore, it is important to achieve differentiated enhancement of low-light and high-light areas. In this paper, we propose a network called correcting uneven illumination network (CUI-Net) with sparse attention transformer and convolutional neural network (CNN) to better extract low-light features by constraining high-light features. Specifically, CUI-Net consists of two main modules: a low-light enhancement module and an auxiliary module. The enhancement module is a hybrid network that combines the advantages of CNN and Transformer network, which can alleviate uneven lighting problems and enhance local details better. The auxiliary module is used to converge the enhancement results of multiple enhancement modules during the training phase, so that only one enhancement module is needed during the testing phase to speed up inference. Furthermore, zero-shot learning is used in this paper to adapt to complex uneven lighting environments without requiring paired or unpaired training data. Finally, to validate the effectiveness of the algorithm, we tested it on multiple datasets of different types, and the algorithm showed stable performance, demonstrating its good robustness. Additionally, by applying this algorithm to practical visual tasks such as object detection, face detection, and semantic segmentation, and comparing it with other state-of-the-art low-light image enhancement algorithms, we have demonstrated its practicality and advantages.
Collapse
Affiliation(s)
- Ke Chao
- School of Information Engineering, Minzu University of China, Beijing, 100081, China
| | - Wei Song
- School of Information Engineering, Minzu University of China, Beijing, 100081, China.
- Key Laboratory of Marine Environmental Survey Technology and Application, Ministry of Natural Resource, Guangzhou, 510300, China.
- Language Information Security Research Center, Institute of National Security MUC, Minzu University of China, Beijing, 100081, China.
- National Language Resource Monitoring and Research Center of Minority Languages, Minzu University of China, Beijing, 100081, China.
- Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China, Beijing, China.
| | - Sen Shao
- School of Information Engineering, Minzu University of China, Beijing, 100081, China
| | - Dan Liu
- School of Information Engineering, Minzu University of China, Beijing, 100081, China
| | - Xiangchun Liu
- School of Information Engineering, Minzu University of China, Beijing, 100081, China
| | - XiaoBing Zhao
- School of Information Engineering, Minzu University of China, Beijing, 100081, China
- Language Information Security Research Center, Institute of National Security MUC, Minzu University of China, Beijing, 100081, China
- National Language Resource Monitoring and Research Center of Minority Languages, Minzu University of China, Beijing, 100081, China
- Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China, Beijing, China
| |
Collapse
|
14
|
Liu R, Ma L, Ma T, Fan X, Luo Z. Learning With Nested Scene Modeling and Cooperative Architecture Search for Low-Light Vision. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:5953-5969. [PMID: 36215366 DOI: 10.1109/tpami.2022.3212995] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
Images captured from low-light scenes often suffer from severe degradations, including low visibility, color casts, intensive noises, etc. These factors not only degrade image qualities, but also affect the performance of downstream Low-Light Vision (LLV) applications. A variety of deep networks have been proposed to enhance the visual quality of low-light images. However, they mostly rely on significant architecture engineering and often suffer from the high computational burden. More importantly, it still lacks an efficient paradigm to uniformly handle various tasks in the LLV scenarios. To partially address the above issues, we establish Retinex-inspired Unrolling with Architecture Search (RUAS), a general learning framework, that can address low-light enhancement task, and has the flexibility to handle other challenging downstream vision tasks. Specifically, we first establish a nested optimization formulation, together with an unrolling strategy, to explore underlying principles of a series of LLV tasks. Furthermore, we design a differentiable strategy to cooperatively search specific scene and task architectures for RUAS. Last but not least, we demonstrate how to apply RUAS for both low- and high-level LLV applications (e.g., enhancement, detection and segmentation). Extensive experiments verify the flexibility, effectiveness, and efficiency of RUAS.
Collapse
|
15
|
An adaptive image enhancement approach for safety monitoring robot under insufficient illumination condition. COMPUT IND 2023. [DOI: 10.1016/j.compind.2023.103862] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
|
16
|
Ali Z, Naz S, Zaffar H, Choi J, Kim Y. An IoMT-Based Melanoma Lesion Segmentation Using Conditional Generative Adversarial Networks. SENSORS (BASEL, SWITZERLAND) 2023; 23:3548. [PMID: 37050607 PMCID: PMC10098854 DOI: 10.3390/s23073548] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/20/2023] [Revised: 02/03/2023] [Accepted: 03/25/2023] [Indexed: 06/19/2023]
Abstract
Currently, Internet of medical things-based technologies provide a foundation for remote data collection and medical assistance for various diseases. Along with developments in computer vision, the application of Artificial Intelligence and Deep Learning in IOMT devices aids in the design of effective CAD systems for various diseases such as melanoma cancer even in the absence of experts. However, accurate segmentation of melanoma skin lesions from images by CAD systems is necessary to carry out an effective diagnosis. Nevertheless, the visual similarity between normal and melanoma lesions is very high, which leads to less accuracy of various traditional, parametric, and deep learning-based methods. Hence, as a solution to the challenge of accurate segmentation, we propose an advanced generative deep learning model called the Conditional Generative Adversarial Network (cGAN) for lesion segmentation. In the suggested technique, the generation of segmented images is conditional on dermoscopic images of skin lesions to generate accurate segmentation. We assessed the proposed model using three distinct datasets including DermQuest, DermIS, and ISCI2016, and attained optimal segmentation results of 99%, 97%, and 95% performance accuracy, respectively.
Collapse
Affiliation(s)
- Zeeshan Ali
- R & D Setups, National University of Computer and Emerging Sciences, Islamabad 44000, Pakistan
| | - Sheneela Naz
- Department of Computer Science, COMSATS University Islamabad, Islamabad 45550, Pakistan
| | - Hira Zaffar
- Department of Computer Science, Air University, Aerospace and Aviation Kamra Campus, Islamabad 44000, Pakistan
| | - Jaeun Choi
- College of Business, Kwangwoon University, Seoul 01897, Republic of Korea
| | - Yongsung Kim
- Department of Technology Education, Chungnam National University, Daejeon 34134, Republic of Korea
| |
Collapse
|
17
|
Li L, Li D, Wang S, Jiao Q, Bian L. Tuning-free and self-supervised image enhancement against ill exposure. OPTICS EXPRESS 2023; 31:10368-10385. [PMID: 37157585 DOI: 10.1364/oe.484628] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/10/2023]
Abstract
Complex lighting conditions and the limited dynamic range of imaging devices result in captured images with ill exposure and information loss. Existing image enhancement methods based on histogram equalization, Retinex-inspired decomposition model, and deep learning suffer from manual tuning or poor generalization. In this work, we report an image enhancement method against ill exposure with self-supervised learning, enabling tuning-free correction. First, a dual illumination estimation network is constructed to estimate the illumination for under- and over-exposed areas. Thus, we get the corresponding intermediate corrected images. Second, given the intermediate corrected images with different best-exposed areas, Mertens' multi-exposure fusion strategy is utilized to fuse the intermediate corrected images to acquire a well-exposed image. The correction-fusion manner allows adaptive dealing with various types of ill-exposed images. Finally, the self-supervised learning strategy is studied which learns global histogram adjustment for better generalization. Compared to training on paired datasets, we only need ill-exposed images. This is crucial in cases where paired data is inaccessible or less than perfect. Experiments show that our method can reveal more details with better visual perception than other state-of-the-art methods. Furthermore, the weighted average scores of image naturalness matrics NIQE and BRISQUE, and contrast matrics CEIQ and NSS on five real-world image datasets are boosted by 7%, 15%, 4%, and 2%, respectively, over the recent exposure correction method.
Collapse
|
18
|
Wang W, Wang X, Yang W, Liu J. Unsupervised Face Detection in the Dark. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:1250-1266. [PMID: 35180078 DOI: 10.1109/tpami.2022.3152562] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Low-light face detection is challenging but critical for real-world applications, such as nighttime autonomous driving and city surveillance. Current face detection models rely on extensive annotations and lack generality and flexibility. In this paper, we explore how to learn face detectors without low-light annotations. Fully exploiting existing normal light data, we propose adapting face detectors from normal light to low light. This task is difficult because the gap between brightness and darkness is too large and complicated at the object level and pixel level. Accordingly, the performance of current low-light enhancement or adaptation methods is unsatisfactory. To solve this problem, we propose a joint High-Low Adaptation (HLA) framework. We design bidirectional low-level adaptation and multitask high-level adaptation. For low-level, we enhance the dark images and degrade the normal-light images, making both domains move toward each other. For high-level, we combine context-based and contrastive learning to comprehensively close the features on different domains. Experiments show that our HLA-Face v2 model obtains superior low-light face detection performance even without the use of low-light annotations. Moreover, our adaptation scheme can be extended to a wide range of applications, such as improving supervised learning and generic object detection. Project publicly available at: https://daooshee.github.io/HLA-Face-v2-Website/.
Collapse
|
19
|
Wang X, Piao Y, Wang Y. A dark image enhancement method based on multiscale features and dilated residual networks. Neural Process Lett 2022. [DOI: 10.1007/s11063-022-10872-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
|
20
|
Hodges C, Bennamoun M, Boussaid F. Quantitative performance evaluation of object detectors in hazy environments. Pattern Recognit Lett 2021. [DOI: 10.1016/j.patrec.2021.10.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
|
21
|
Zuo Z, Watson M, Budgen D, Hall R, Kennelly C, Al Moubayed N. Data Anonymization for Pervasive Health Care: Systematic Literature Mapping Study. JMIR Med Inform 2021; 9:e29871. [PMID: 34652278 PMCID: PMC8556642 DOI: 10.2196/29871] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2021] [Revised: 06/21/2021] [Accepted: 08/02/2021] [Indexed: 01/29/2023] Open
Abstract
BACKGROUND Data science offers an unparalleled opportunity to identify new insights into many aspects of human life with recent advances in health care. Using data science in digital health raises significant challenges regarding data privacy, transparency, and trustworthiness. Recent regulations enforce the need for a clear legal basis for collecting, processing, and sharing data, for example, the European Union's General Data Protection Regulation (2016) and the United Kingdom's Data Protection Act (2018). For health care providers, legal use of the electronic health record (EHR) is permitted only in clinical care cases. Any other use of the data requires thoughtful considerations of the legal context and direct patient consent. Identifiable personal and sensitive information must be sufficiently anonymized. Raw data are commonly anonymized to be used for research purposes, with risk assessment for reidentification and utility. Although health care organizations have internal policies defined for information governance, there is a significant lack of practical tools and intuitive guidance about the use of data for research and modeling. Off-the-shelf data anonymization tools are developed frequently, but privacy-related functionalities are often incomparable with regard to use in different problem domains. In addition, tools to support measuring the risk of the anonymized data with regard to reidentification against the usefulness of the data exist, but there are question marks over their efficacy. OBJECTIVE In this systematic literature mapping study, we aim to alleviate the aforementioned issues by reviewing the landscape of data anonymization for digital health care. METHODS We used Google Scholar, Web of Science, Elsevier Scopus, and PubMed to retrieve academic studies published in English up to June 2020. Noteworthy gray literature was also used to initialize the search. We focused on review questions covering 5 bottom-up aspects: basic anonymization operations, privacy models, reidentification risk and usability metrics, off-the-shelf anonymization tools, and the lawful basis for EHR data anonymization. RESULTS We identified 239 eligible studies, of which 60 were chosen for general background information; 16 were selected for 7 basic anonymization operations; 104 covered 72 conventional and machine learning-based privacy models; four and 19 papers included seven and 15 metrics, respectively, for measuring the reidentification risk and degree of usability; and 36 explored 20 data anonymization software tools. In addition, we also evaluated the practical feasibility of performing anonymization on EHR data with reference to their usability in medical decision-making. Furthermore, we summarized the lawful basis for delivering guidance on practical EHR data anonymization. CONCLUSIONS This systematic literature mapping study indicates that anonymization of EHR data is theoretically achievable; yet, it requires more research efforts in practical implementations to balance privacy preservation and usability to ensure more reliable health care applications.
Collapse
Affiliation(s)
- Zheming Zuo
- Department of Computer Science, Durham University, Durham, United Kingdom
| | - Matthew Watson
- Department of Computer Science, Durham University, Durham, United Kingdom
| | - David Budgen
- Department of Computer Science, Durham University, Durham, United Kingdom
| | - Robert Hall
- Cievert Ltd, Newcastle upon Tyne, United Kingdom
| | | | - Noura Al Moubayed
- Department of Computer Science, Durham University, Durham, United Kingdom
| |
Collapse
|
22
|
Ngo D, Lee S, Ngo TM, Lee GD, Kang B. Visibility Restoration: A Systematic Review and Meta-Analysis. SENSORS 2021; 21:s21082625. [PMID: 33918021 PMCID: PMC8069147 DOI: 10.3390/s21082625] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/03/2021] [Revised: 03/29/2021] [Accepted: 04/06/2021] [Indexed: 11/16/2022]
Abstract
Image acquisition is a complex process that is affected by a wide variety of internal and environmental factors. Hence, visibility restoration is crucial for many high-level applications in photography and computer vision. This paper provides a systematic review and meta-analysis of visibility restoration algorithms with a focus on those that are pertinent to poor weather conditions. This paper starts with an introduction to optical image formation and then provides a comprehensive description of existing algorithms as well as a comparative evaluation. Subsequently, there is a thorough discussion on current difficulties that are worthy of a scientific effort. Moreover, this paper proposes a general framework for visibility restoration in hazy weather conditions while using haze-relevant features and maximum likelihood estimates. Finally, a discussion on the findings and future developments concludes this paper.
Collapse
Affiliation(s)
- Dat Ngo
- Department of Electronics Engineering, Dong-A University, Busan 49315, Korea; (D.N.); (S.L.); (G.-D.L.)
| | - Seungmin Lee
- Department of Electronics Engineering, Dong-A University, Busan 49315, Korea; (D.N.); (S.L.); (G.-D.L.)
| | - Tri Minh Ngo
- Faculty of Electronics and Telecommunication Engineering, The University of Danang—University of Science and Technology, Danang 550000, Vietnam;
| | - Gi-Dong Lee
- Department of Electronics Engineering, Dong-A University, Busan 49315, Korea; (D.N.); (S.L.); (G.-D.L.)
| | - Bongsoon Kang
- Department of Electronics Engineering, Dong-A University, Busan 49315, Korea; (D.N.); (S.L.); (G.-D.L.)
- Correspondence: ; Tel.: +82-51-200-7703
| |
Collapse
|
23
|
Yang W, Wang S, Fang Y, Wang Y, Liu J. Band Representation-Based Semi-Supervised Low-Light Image Enhancement: Bridging the Gap Between Signal Fidelity and Perceptual Quality. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2021; 30:3461-3473. [PMID: 33656992 DOI: 10.1109/tip.2021.3062184] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
It has been widely acknowledged that under-exposure causes a variety of visual quality degradation because of intensive noise, decreased visibility, biased color, etc. To alleviate these issues, a novel semi-supervised learning approach is proposed in this paper for low-light image enhancement. More specifically, we propose a deep recursive band network (DRBN) to recover a linear band representation of an enhanced normal-light image based on the guidance of the paired low/normal-light images. Such design philosophy enables the principled network to generate a quality improved one by reconstructing the given bands based upon another learnable linear transformation which is perceptually driven by an image quality assessment neural network. On one hand, the proposed network is delicately developed to obtain a variety of coarse-to-fine band representations, of which the estimations benefit each other in a recursive process mutually. On the other hand, the extracted band representation of the enhanced image in the recursive band learning stage of DRBN is capable of bridging the gap between the restoration knowledge of paired data and the perceptual quality preference to high-quality images. Subsequently, the band recomposition learns to recompose the band representation towards fitting perceptual regularization of high-quality images with the perceptual guidance. The proposed architecture can be flexibly trained with both paired and unpaired data. Extensive experiments demonstrate that our method produces better enhanced results with visually pleasing contrast and color distributions, as well as well-restored structural details.
Collapse
|
24
|
Jiang Y, Gong X, Liu D, Cheng Y, Fang C, Shen X, Yang J, Zhou P, Wang Z. EnlightenGAN: Deep Light Enhancement Without Paired Supervision. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2021; 30:2340-2349. [PMID: 33481709 DOI: 10.1109/tip.2021.3051462] [Citation(s) in RCA: 146] [Impact Index Per Article: 36.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Deep learning-based methods have achieved remarkable success in image restoration and enhancement, but are they still competitive when there is a lack of paired training data? As one such example, this paper explores the low-light image enhancement problem, where in practice it is extremely challenging to simultaneously take a low-light and a normal-light photo of the same visual scene. We propose a highly effective unsupervised generative adversarial network, dubbed EnlightenGAN, that can be trained without low/normal-light image pairs, yet proves to generalize very well on various real-world test images. Instead of supervising the learning using ground truth data, we propose to regularize the unpaired training using the information extracted from the input itself, and benchmark a series of innovations for the low-light image enhancement problem, including a global-local discriminator structure, a self-regularized perceptual loss fusion, and the attention mechanism. Through extensive experiments, our proposed approach outperforms recent methods under a variety of metrics in terms of visual quality and subjective user study. Thanks to the great flexibility brought by unpaired training, EnlightenGAN is demonstrated to be easily adaptable to enhancing real-world images from various domains. Our codes and pre-trained models are available at: https://github.com/VITA-Group/EnlightenGAN.
Collapse
|