Interactive residual coordinate attention and contrastive learning for infrared and visible image fusion in triple frequency bands.
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
02 Jan 2024
02 Jan 2024
Historique:
received:
03
10
2023
accepted:
29
12
2023
medline:
4
1
2024
pubmed:
4
1
2024
entrez:
3
1
2024
Statut:
epublish
Résumé
The auto-encoder (AE) based image fusion models have achieved encouraging performance on infrared and visible image fusion. However, the meaningful information loss in the encoding stage and simple unlearnable fusion strategy are two significant challenges for such models. To address these issues, this paper proposes an infrared and visible image fusion model based on interactive residual attention fusion strategy and contrastive learning in the frequency domain. Firstly, the source image is transformed into three sub-bands of the high-frequency, low-frequency, and mid-frequency for powerful multiscale representation from the prospective of the frequency spectrum analysis. To further cope with the limitations of the straightforward fusion strategy, a learnable coordinate attention module in the fusion layer is incorporated to adaptively fuse representative information based on the characteristics of the corresponding feature maps. Moreover, the contrastive learning is leveraged to train the multiscale decomposition network for enhancing the complementarity of information at different frequency spectra. Finally, the detail-preserving loss, feature enhancing loss and contrastive loss are incorporated to jointly train the entire fusion model for good detail maintainability. Qualitative and quantitative comparisons demonstrate the feasibility and validity of our model, which can consistently generate fusion images containing both highlight targets and legible details, outperforming the state-of-the-art fusion methods.
Identifiants
pubmed: 38167638
doi: 10.1038/s41598-023-51045-9
pii: 10.1038/s41598-023-51045-9
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
90Subventions
Organisme : National Nature Science Foundation of China
ID : 62362037, 12264018
Informations de copyright
© 2024. The Author(s).
Références
Ma, J., Ma, Y. & Li, C. Infrared and visible image fusion methods and applications: A survey. Inf. Fusion 45, 153–178 (2019).
doi: 10.1016/j.inffus.2018.02.004
Zou, D. & Yang, B. Infrared and low-light visible image fusion based on hybrid multiscale decomposition and adaptive light adjustment. Opt. Lasers Eng. 160, 107268 (2022).
doi: 10.1016/j.optlaseng.2022.107268
Liu, Y. et al. Infrared and visible image fusion with convolutional neural networks. Int. J. Wavelets Multiresolut. Inf. Process. 16(03), 1850018 (2018).
doi: 10.1142/S0219691318500182
Xu, H., Zhang, H. & Ma, J. Classification saliency-based rule for visible and infrared image fusion. IEEE Trans. Comput. Imaging 7, 824–836 (2021).
doi: 10.1109/TCI.2021.3100986
Liu, L. et al. Two-stream network for infrared and visible images fusion. Neurocomputing 460, 50–58 (2021).
doi: 10.1016/j.neucom.2021.05.034
Wang, B. et al. Multimodal super-resolution reconstruction of infrared and visible images via deep learning. Opt. Lasers Eng. 156, 107078 (2022).
doi: 10.1016/j.optlaseng.2022.107078
Ma, J. et al. STDFusionNet: An infrared and visible image fusion network based on salient target detection. IEEE Trans. Instrum. Meas. 70, 1–13 (2021).
Song, X. et al. Res2NetFuse: A Fusion Method for Infrared and Visible Images. arXiv preprint arXiv:2112.14540 (2021).
Goodfellow, I. et al. Generative adversarial nets. Adv. Neural Inf. Process. Syst. 27 (2014).
Ma, J. et al. FusionGAN: A generative adversarial network for infrared and visible image fusion. Inf. Fusion 48, 11–26 (2019).
doi: 10.1016/j.inffus.2018.09.004
Ma, J. et al. GANMcC: A generative adversarial network with multiclassification constraints for infrared and visible image fusion. IEEE Trans. Instrum. Meas. 70, 1–14 (2020).
Li, H., Wu, X. J. & Kittler, J. RFN-Nest: An end-to-end residual fusion network for infrared and visible images. Inf. Fusion 73, 72–86 (2021).
doi: 10.1016/j.inffus.2021.02.023
Li, K., Lu, J. & Zuo, H. Dynamic classifier alignment for unsupervised multi-source domain adaptation. IEEE Trans. Knowl. Data Eng. 35(5), 4727–4740 (2023).
Cao, F. & Li, K. A new method for image super-resolution with multi-channel constraints. Knowl.-Based Syst. 146, 118–128 (2018).
doi: 10.1016/j.knosys.2018.01.034
Zhao, Z. et al. Efficient and model-based infrared and visible image fusion via algorithm unrolling. IEEE Trans. Circuits Syst. Video Technol. 32(3), 1186–1196 (2022).
doi: 10.1109/TCSVT.2021.3075745
Song, X. & Wu, X. J. Multi-focus Image Fusion with PCA Filters of PCANet. IAPR Workshop on Multimodal Pattern Recognition of Social Signals in Human-Computer Interaction 1–17 (Springer, 2018).
Li, H. & Wu, X. J. DenseFuse: A fusion approach to infrared and visible images. IEEE Trans. Image Process. 28(5), 2614–2623 (2018).
doi: 10.1109/TIP.2018.2887342
Ram Prabhakar, K., Sai Srikar, V. & Venkatesh Babu, R. Deepfuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs. Proceedings of the IEEE International Conference on Computer Vision 4714–4722 (2017).
Bulanon, D. M., Burks, T. F. & Alchanatis, V. Image fusion of visible and thermal images for fruit detection. Biosyst. Eng. 103(1), 12–22 (2009).
doi: 10.1016/j.biosystemseng.2009.02.009
Jin, X. et al. Infrared and visual image fusion method based on discrete cosine transform and local spatial frequency in discrete stationary wavelet transform domain. Infrared Phys. Technol. 88, 1–12 (2018).
doi: 10.1016/j.infrared.2017.10.004
Xiang, T., Yan, L. & Gao, R. A fusion algorithm for infrared and visible images based on adaptive dual-channel unit-linking PCNN in NSCT domain. Infrared Phys. Technol. 69, 53–61 (2015).
doi: 10.1016/j.infrared.2015.01.002
Hu, J. & Li, S. The multiscale directional bilateral filter and its application to multisensor image fusion. Inf. Fusion 13(3), 196–206 (2012).
doi: 10.1016/j.inffus.2011.01.002
Jian, L. et al. SEDRFuse: A symmetric encoder–decoder with residual block network for infrared and visible image fusion. IEEE Trans. Instrum. Meas. 70, 1–15 (2020).
doi: 10.1109/TIM.2020.3022438
Long, Y. et al. RXDNFuse: A aggregated residual dense network for infrared and visible image fusion. Inf. Fusion 69, 128–141 (2021).
doi: 10.1016/j.inffus.2020.11.009
Zhu, Z. et al. A novel multi-modality image fusion method based on image decomposition and sparse representation. Inf. Sci. 432, 516–529 (2018).
doi: 10.1016/j.ins.2017.09.010
Kim, M., Han, D. K. & Ko, H. Joint patch clustering-based dictionary learning for multimodal image fusion. Inf. Fusion 27, 198–214 (2016).
doi: 10.1016/j.inffus.2015.03.003
Zhang, H. et al. Rethinking the image fusion: A fast unified image fusion network based on proportional maintenance of gradient and intensity. Proc. AAAI Conf. Artif. Intell. 34(07), 12797–12804 (2020).
Tang, L., Yuan, J. & Ma, J. Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network. Information Fusion 82, 28–42 (2022).
doi: 10.1016/j.inffus.2021.12.004
Zhang, H. et al. EPSANet: An Efficient Pyramid Squeeze Attention Block on Convolutional Neural Network. arXiv preprint arXiv:2105.14447 (2021).
Hou, Q., Zhou, D. & Feng, J. Coordinate attention for efficient mobile network design. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 13713–13722 (2021).
Wang, Z. et al. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004).
doi: 10.1109/TIP.2003.819861
pubmed: 15376593
Toet A. “TNO Image Fusion Dataset,” 4 2014. https://figshare.com/articles/dataset/TNOImageFusionDataset/1008029 .
Xu, H. et al. Fusiondn: A unified densely connected network for image fusion. Proc. AAAI Conf. Artif. Intell. 34(7), 12484–12491 (2020).
Brown, M. & Süsstrunk, S. Multi-spectral SIFT for scene category recognition. CVPR 2011 177–184 (IEEE, 2011).
Zhao, Z. et al. DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion. arXiv preprint arXiv:2003.09210 , 2020.
Li, H., Wu, X. J. & Durrani, T. NestFuse: An infrared and visible image fusion architecture based on nest connection and spatial/channel attention models. IEEE Trans. Instrum. Meas. 69(12), 9645–9656 (2020).
doi: 10.1109/TIM.2020.3005230
Fu, Y. & Wu, X. J. A dual-branch network for infrared and visible image fusion. 2020 25th International Conference on Pattern Recognition (ICPR) 10675–10680 (IEEE, 2021).
Wang, Z. et al. UNFusion: A unified multi-scale densely connected network for infrared and visible image fusion. IEEE Trans. Circuits Syst. Video Technol. 32(6), 3360–3374 (2022).
doi: 10.1109/TCSVT.2021.3109895
Wang, Z. et al. Res2Fusion: Infrared and visible image fusion based on dense Res2net and double non-local attention models. IEEE Trans. Instrum. Meas. 71, 1–16 (2022).
doi: 10.1109/TIM.2022.3216413
Xu, H. et al. CUFD: An encoder–decoder network for visible and infrared image fusion based on common and unique feature decomposition. Comput. Vis. Image Understand. 218, 103407 (2022).
doi: 10.1016/j.cviu.2022.103407
Li, H. et al. LRRNet: A novel representation learning guided fusion network for infrared and visible images. IEEE Trans. Pattern Anal. Mach. Intell. 45(9), 11040–11052 (2023).
doi: 10.1109/TPAMI.2023.3268209
pubmed: 37074897
Zhao, Z. et al. Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 5906–5916 (2023).
Tang, L. et al. Rethinking the necessity of image fusion in high-level vision tasks: A practical infrared and visible image fusion network based on progressive semantic injection and scene fidelity. Inf. Fusion 99, 101870 (2023).
doi: 10.1016/j.inffus.2023.101870