Drop Clause: Enhancing Performance, Robustness and Pattern Recognition Capabilities of the Tsetlin Machine

Authors

  • Jivitesh Sharma University of Agder
  • Rohan Yadav University of Agder
  • Ole-Christoffer Granmo University of Agder
  • Lei Jiao University of Agder

DOI:

https://doi.org/10.1609/aaai.v37i11.26588

Keywords:

SNLP: Interpretability & Analysis of NLP Models, KRR: Logic Programming, ML: Adversarial Learning & Robustness, ML: Distributed Machine Learning & Federated Learning, ML: Ensemble Methods, ML: Optimization, SNLP: Sentence-Level Semantics and Textual Inference

Abstract

Logic-based machine learning has the crucial advantage of transparency. However, despite significant recent progress, further research is needed to close the accuracy gap between logic-based architectures and deep neural network ones. This paper introduces a novel variant of the Tsetlin machine (TM) that randomly drops clauses, the logical learning element of TMs. In effect, TM with Drop Clause ignores a random selection of the clauses in each epoch, selected according to a predefined probability. In this way, the TM learning phase becomes more diverse. To explore the effects that Drop Clause has on accuracy, training time and robustness, we conduct extensive experiments on nine benchmark datasets in natural language processing (IMDb, R8, R52, MR, and TREC) and image classification (MNIST, Fashion MNIST, CIFAR-10, and CIFAR-100). Our proposed model outperforms baseline machine learning algorithms by a wide margin and achieves competitive performance compared with recent deep learning models, such as BERT-Large and AlexNet-DFA. In brief, we observe up to +10% increase in accuracy and 2x to 4x faster learning than for the standard TM. We visualize the patterns learnt by Drop Clause TM in the form of heatmaps and show evidence of the ability of drop clause to learn more unique and discriminative patterns. We finally evaluate how Drop Clause affects learning robustness by introducing corruptions and alterations in the image/language test data, which exposes increased learning robustness.

Downloads

Published

2023-06-26

How to Cite

Sharma, J., Yadav, R., Granmo, O.-C., & Jiao, L. (2023). Drop Clause: Enhancing Performance, Robustness and Pattern Recognition Capabilities of the Tsetlin Machine. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11), 13547-13555. https://doi.org/10.1609/aaai.v37i11.26588

Issue

Section

AAAI Technical Track on Speech & Natural Language Processing