Fractional Adaptive Linear Units
DOI:
https://doi.org/10.1609/aaai.v36i8.20882Keywords:
Machine Learning (ML)Abstract
This work introduces Fractional Adaptive Linear Units (FALUs), a flexible generalization of adaptive activation functions. Leveraging principles from fractional calculus, FALUs define a diverse family of activation functions (AFs) that encompass many traditional and state-of-the-art activation functions. This family includes the Sigmoid, Gaussian, ReLU, GELU, and Swish functions, as well as a large variety of smooth interpolations between these functions. Our technique requires only a small number of additional trainable parameters, and needs no further specialized optimization or initialization procedures. For this reason, FALUs present a seamless and rich automated solution to the problem of activation function optimization. Through experiments on a variety of conventional tasks and network architectures, we demonstrate the effectiveness of FALUs when compared to traditional and state-of-the-art AFs. To facilitate practical use of this work, we plan to make our code publicly availableDownloads
Published
2022-06-28
How to Cite
Zamora, J., Rhodes, A. D., & Nachman, L. (2022). Fractional Adaptive Linear Units. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8), 8988-8996. https://doi.org/10.1609/aaai.v36i8.20882
Issue
Section
AAAI Technical Track on Machine Learning III