SiamTrans: Zero-Shot Multi-Frame Image Restoration with Pre-trained Siamese Transformers
Keywords:Computer Vision (CV)
AbstractWe propose a novel zero-shot multi-frame image restoration method for removing unwanted obstruction elements (such as rains, snow, and moire patterns) that vary in successive frames. It has three stages: transformer pre-training, zero-shot restoration, and hard patch refinement. Using the pre-trained transformers, our model is able to tell the motion difference between the true image information and the obstructing elements. For zero-shot image restoration, we design a novel model, termed SiamTrans, which is constructed by Siamese transformers, encoders, and decoders. Each transformer has a temporal attention layer and several self-attention layers, to capture both temporal and spatial information of multiple frames. Only self-supervisedly pre-trained on the denoising task, SiamTrans is tested on three different low-level vision tasks (deraining, demoireing, and desnowing). Compared with related methods, SiamTrans achieves the best performances, even outperforming those with supervised learning.
How to Cite
Liu, L., Yuan, S., Liu, J., Guo, X., Yan, Y., & Tian, Q. (2022). SiamTrans: Zero-Shot Multi-Frame Image Restoration with Pre-trained Siamese Transformers. Proceedings of the AAAI Conference on Artificial Intelligence, 36(2), 1747-1755. https://doi.org/10.1609/aaai.v36i2.20067
AAAI Technical Track on Computer Vision II