Multi-Modality Deep Network for Extreme Learned Image Compression
DOI:
https://doi.org/10.1609/aaai.v37i1.25184Keywords:
CV: Multi-modal Vision, CV: Low Level & Physics-Based VisionAbstract
Image-based single-modality compression learning approaches have demonstrated exceptionally powerful encoding and decoding capabilities in the past few years , but suffer from blur and severe semantics loss at extremely low bitrates. To address this issue, we propose a multimodal machine learning method for text-guided image compression, in which the semantic information of text is used as prior information to guide image compression for better compression performance. We fully study the role of text description in different components of the codec, and demonstrate its effectiveness. In addition, we adopt the image-text attention module and image-request complement module to better fuse image and text features, and propose an improved multimodal semantic-consistent loss to produce semantically complete reconstructions. Extensive experiments, including a user study, prove that our method can obtain visually pleasing results at extremely low bitrates, and achieves a comparable or even better performance than state-of-the-art methods, even though these methods are at 2x to 4x bitrates of ours.Downloads
Published
2023-06-26
How to Cite
Jiang, X., Tan, W., Tan, T., Yan, B., & Shen, L. (2023). Multi-Modality Deep Network for Extreme Learned Image Compression. Proceedings of the AAAI Conference on Artificial Intelligence, 37(1), 1033-1041. https://doi.org/10.1609/aaai.v37i1.25184
Issue
Section
AAAI Technical Track on Computer Vision I