Machines of Finite Depth: Towards a Formalization of Neural Networks - Vertechi, Pietro; Bergomi, Mattia G. (Proceedings of the AAAI Conference on Artificial Intelligence, 2023)
We provide a unifying framework where artificial neural networks and their architectures can be formally described as particular cases of a general mathematical construction—machines of finite depth. Unlike neural networks, machines have a precise definition, from which several properties follow naturally. Machines of finite depth are modular (they can be combined), efficiently computable, and differentiable. The backward pass of a machine is again a machine and can be computed without overhead using the same procedure as the forward pass. We prove this statement theoretically and practically via a unified implementation that generalizes several classical architectures—dense, convolutional, and recurrent neural networks with a rich shortcut structure—and their respective backpropagation rules.