Scaling Up Influence Functions

Authors

  • Andrea Schioppa Google
  • Polina Zablotskaia Google
  • David Vilar Google
  • Artem Sokolov Google

DOI:

https://doi.org/10.1609/aaai.v36i8.20791

Keywords:

Machine Learning (ML), Speech & Natural Language Processing (SNLP), Computer Vision (CV), Data Mining & Knowledge Management (DMKM)

Abstract

We address efficient calculation of influence functions for tracking predictions back to the training data. We propose and analyze a new approach to speeding up the inverse Hessian calculation based on Arnoldi iteration. With this improvement, we achieve, to the best of our knowledge, the first successful implementation of influence functions that scales to full-size (language and vision) Transformer models with several hundreds of millions of parameters. We evaluate our approach in image classification and sequence-to-sequence tasks with tens to a hundred of millions of training examples. Our code is available at https://github.com/google-research/jax-influence.

Downloads

Published

2022-06-28

How to Cite

Schioppa, A., Zablotskaia, P., Vilar, D., & Sokolov, A. (2022). Scaling Up Influence Functions. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8), 8179-8186. https://doi.org/10.1609/aaai.v36i8.20791

Issue

Section

AAAI Technical Track on Machine Learning III