SiamTrans: Zero-Shot Multi-Frame Image Restoration with Pre-trained Siamese Transformers
DOI:
https://doi.org/10.1609/aaai.v36i2.20067Keywords:
Computer Vision (CV)Abstract
We propose a novel zero-shot multi-frame image restoration method for removing unwanted obstruction elements (such as rains, snow, and moire patterns) that vary in successive frames. It has three stages: transformer pre-training, zero-shot restoration, and hard patch refinement. Using the pre-trained transformers, our model is able to tell the motion difference between the true image information and the obstructing elements. For zero-shot image restoration, we design a novel model, termed SiamTrans, which is constructed by Siamese transformers, encoders, and decoders. Each transformer has a temporal attention layer and several self-attention layers, to capture both temporal and spatial information of multiple frames. Only self-supervisedly pre-trained on the denoising task, SiamTrans is tested on three different low-level vision tasks (deraining, demoireing, and desnowing). Compared with related methods, SiamTrans achieves the best performances, even outperforming those with supervised learning.Downloads
Published
2022-06-28
How to Cite
Liu, L., Yuan, S., Liu, J., Guo, X., Yan, Y., & Tian, Q. (2022). SiamTrans: Zero-Shot Multi-Frame Image Restoration with Pre-trained Siamese Transformers. Proceedings of the AAAI Conference on Artificial Intelligence, 36(2), 1747-1755. https://doi.org/10.1609/aaai.v36i2.20067
Issue
Section
AAAI Technical Track on Computer Vision II