Hybrid CNN-Transformer Feature Fusion for Single Image Deraining


  • Xiang Chen Nanjing University of Science and Technology
  • Jinshan Pan Nanjing University of Science and Technology
  • Jiyang Lu Shenyang Aerospace University
  • Zhentao Fan Shenyang Aerospace University
  • Hao Li Nanjing University of Science and Technology




CV: Low Level & Physics-Based Vision, CV: Applications, CV: Computational Photography, Image & Video Synthesis, CV: Representation Learning for Vision


Since rain streaks exhibit diverse geometric appearances and irregular overlapped phenomena, these complex characteristics challenge the design of an effective single image deraining model. To this end, rich local-global information representations are increasingly indispensable for better satisfying rain removal. In this paper, we propose a lightweight Hybrid CNN-Transformer Feature Fusion Network (dubbed as HCT-FFN) in a stage-by-stage progressive manner, which can harmonize these two architectures to help image restoration by leveraging their individual learning strengths. Specifically, we stack a sequence of the degradation-aware mixture of experts (DaMoE) modules in the CNN-based stage, where appropriate local experts adaptively enable the model to emphasize spatially-varying rain distribution features. As for the Transformer-based stage, a background-aware vision Transformer (BaViT) module is employed to complement spatially-long feature dependencies of images, so as to achieve global texture recovery while preserving the required structure. Considering the indeterminate knowledge discrepancy among CNN features and Transformer features, we introduce an interactive fusion branch at adjacent stages to further facilitate the reconstruction of high-quality deraining results. Extensive evaluations show the effectiveness and extensibility of our developed HCT-FFN. The source code is available at https://github.com/cschenxiang/HCT-FFN.




How to Cite

Chen, X., Pan, J., Lu, J., Fan, Z., & Li, H. (2023). Hybrid CNN-Transformer Feature Fusion for Single Image Deraining. Proceedings of the AAAI Conference on Artificial Intelligence, 37(1), 378-386. https://doi.org/10.1609/aaai.v37i1.25111



AAAI Technical Track on Computer Vision I