DNNs as Layers of Cooperating Classifiers

Authors

  • Marelie Davel North-West University
  • Marthinus Theunissen North-West University
  • Arnold Pretorius North-West University
  • Etienne Barnard North-West University

DOI:

https://doi.org/10.1609/aaai.v34i04.5782

Abstract

A robust theoretical framework that can describe and predict the generalization ability of DNNs in general circumstances remains elusive. Classical attempts have produced complexity metrics that rely heavily on global measures of compactness and capacity with little investigation into the effects of sub-component collaboration. We demonstrate intriguing regularities in the activation patterns of the hidden nodes within fully-connected feedforward networks. By tracing the origin of these patterns, we show how such networks can be viewed as the combination of two information processing systems: one continuous and one discrete. We describe how these two systems arise naturally from the gradient-based optimization process, and demonstrate the classification ability of the two systems, individually and in collaboration. This perspective on DNN classification offers a novel way to think about generalization, in which different subsets of the training data are used to train distinct classifiers; those classifiers are then combined to perform the classification task, and their consistency is crucial for accurate classification.

Downloads

Published

2020-04-03

How to Cite

Davel, M., Theunissen, M., Pretorius, A., & Barnard, E. (2020). DNNs as Layers of Cooperating Classifiers. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04), 3725-3732. https://doi.org/10.1609/aaai.v34i04.5782

Issue

Section

AAAI Technical Track: Machine Learning