Simple and Effective Stochastic Neural Networks

Authors

  • Tianyuan Yu University of Surrey
  • Yongxin Yang University of Surrey
  • Da Li University of Edinburgh Samsung AI Centre
  • Timothy Hospedales University of Edinburgh Samsung AI Centre
  • Tao Xiang University of Surrey

DOI:

https://doi.org/10.1609/aaai.v35i4.16436

Keywords:

Applications

Abstract

Stochastic neural networks (SNNs) are currently topical, with several paradigms being actively investigated including dropout, Bayesian neural networks, variational information bottleneck (VIB) and noise regularized learning. These neural network variants impact several major considerations, including generalization, network compression, robustness against adversarial attack and label noise, and model calibration. However, many existing networks are complicated and expensive to train, and/or only address one or two of these practical considerations. In this paper we propose a simple and effective stochastic neural network (SE-SNN) architecture for discriminative learning by directly modeling activation uncertainty and encouraging high activation variability. Compared to existing SNNs, our SE-SNN is simpler to implement and faster to train, and produces state of the art results on network compression by pruning, adversarial defense, learning with label noise, and model calibration.

Downloads

Published

2021-05-18

How to Cite

Yu, T., Yang, Y., Li, D., Hospedales, T., & Xiang, T. (2021). Simple and Effective Stochastic Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(4), 3252-3260. https://doi.org/10.1609/aaai.v35i4.16436

Issue

Section

AAAI Technical Track on Computer Vision III