TY - JOUR
AU - Lienen, Julian
AU - HÃ¼llermeier, Eyke
PY - 2021/05/18
Y2 - 2023/06/09
TI - From Label Smoothing to Label Relaxation
JF - Proceedings of the AAAI Conference on Artificial Intelligence
JA - AAAI
VL - 35
IS - 10
SE - AAAI Technical Track on Machine Learning III
DO - 10.1609/aaai.v35i10.17041
UR - https://ojs.aaai.org/index.php/AAAI/article/view/17041
SP - 8583-8591
AB - Regularization of (deep) learning models can be realized at the model, loss, or data level. As a technique somewhere in-between loss and data, label smoothing turns deterministic class labels into probability distributions, for example by uniformly distributing a certain part of the probability mass over all classes. A predictive model is then trained on these distributions as targets, using cross-entropy as loss function. While this method has shown improved performance compared to non-smoothed cross-entropy, we argue that the use of a smoothed though still precise probability distribution as a target can be questioned from a theoretical perspective. As an alternative, we propose a generalized technique called label relaxation, in which the target is a set of probabilities represented in terms of an upper probability distribution. This leads to a genuine relaxation of the target instead of a distortion, thereby reducing the risk of incorporating an undesirable bias in the learning process. Methodically, label relaxation leads to the minimization of a novel type of loss function, for which we propose a suitable closed-form expression for model optimization. The effectiveness of the approach is demonstrated in an empirical study on image data.
ER -