Bias-Variance Trade-Off in Hierarchical Probabilistic Models Using Higher-Order Feature Interactions

Authors

  • Simon Luo The University of Sydney
  • Mahito Sugiyama National Institute of Informatics

DOI:

https://doi.org/10.1609/aaai.v33i01.33014488

Abstract

Hierarchical probabilistic models are able to use a large number of parameters to create a model with a high representation power. However, it is well known that increasing the number of parameters also increases the complexity of the model which leads to a bias-variance trade-off. Although it is a classical problem, the bias-variance trade-off between hiddenlayers and higher-order interactions have not been well studied. In our study, we propose an efficient inference algorithm for the log-linear formulation of the higher-order Boltzmann machine using a combination of Gibbs sampling and annealed importance sampling. We then perform a bias-variance decomposition to study the differences in hidden layers and higher-order interactions. Our results have shown that using hidden layers and higher-order interactions have a comparable error with a similar order of magnitude and using higherorder interactions produce less variance for smaller sample size.

Downloads

Published

2019-07-17

How to Cite

Luo, S., & Sugiyama, M. (2019). Bias-Variance Trade-Off in Hierarchical Probabilistic Models Using Higher-Order Feature Interactions. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 4488-4495. https://doi.org/10.1609/aaai.v33i01.33014488

Issue

Section

AAAI Technical Track: Machine Learning