Representing the Unification of Text Featurization using a Context-Free Grammar

Abstract

We propose a novel context-free grammar to represent text embeddings in conjunction with their various transformations. We show how this grammar can serve as a unification layer on top of different featurization techniques, and their hybridization thereof. The approach is embodied in an open-source library, called TextWiser, with a high-level user interface to serve researchers and practitioners. The goal of TextWiser is to enable rapid experimentation with various featurization methods and to serve as a building block within AI applications consuming unstructured data. We highlight several key benefits that are desirable especially in industrial settings where rapid experimentation, reusability, reproducibility, and time to market are of great interest. Finally, we showcase a deployed service powered by TextWiser as a proof-of-concept enterprise application.