TY - JOUR AU - Xue, Chao AU - Wang, Xiaoxing AU - Yan, Junchi AU - Hu, Yonggang AU - Yang, Xiaokang AU - Sun, Kewei PY - 2021/05/18 Y2 - 2024/03/28 TI - Rethinking Bi-Level Optimization in Neural Architecture Search: A Gibbs Sampling Perspective JF - Proceedings of the AAAI Conference on Artificial Intelligence JA - AAAI VL - 35 IS - 12 SE - AAAI Technical Track on Machine Learning V DO - 10.1609/aaai.v35i12.17262 UR - https://ojs.aaai.org/index.php/AAAI/article/view/17262 SP - 10551-10559 AB - One-Shot architecture search, which aims to explore all possible operations jointly based on a single model, has been an active direction of Neural Architecture Search (NAS). As a well-known one-shot solution, Differentiable Architecture Search (DARTS) performs continuous relaxation on the architecture's importance and results in a bi-level optimization problem. However, as many recent studies have shown, DARTS cannot always work robustly for new tasks, which is mainly due to the approximate solution of the bi-level optimization. In this paper, one-shot neural architecture search is addressed by adopting a directed probabilistic graphical model to represent the joint probability distribution over data and model. Then, neural architectures are searched for and optimized by Gibbs sampling. We rethink the bi-level optimization problem as the task of Gibbs sampling from the posterior distribution, which expresses the preferences for different models given the observed dataset. We evaluate our proposed NAS method -- GibbsNAS on the search space used in DARTS/ENAS and the search space of NAS-Bench-201. Experimental results on multiple search space show the efficacy and stability of our approach. ER -