Representing the Unification of Text Featurization using a Context-Free Grammar


  • Doruk Kilitcioglu Fidelity Investments
  • Serdar Kadioglu Fidelity Investments



Text Featurization, Embeddings, Natural Language Processing


We propose a novel context-free grammar to represent text embeddings in conjunction with their various transformations. We show how this grammar can serve as a unification layer on top of different featurization techniques, and their hybridization thereof. The approach is embodied in an open-source library, called TextWiser, with a high-level user interface to serve researchers and practitioners. The goal of TextWiser is to enable rapid experimentation with various featurization methods and to serve as a building block within AI applications consuming unstructured data. We highlight several key benefits that are desirable especially in industrial settings where rapid experimentation, reusability, reproducibility, and time to market are of great interest. Finally, we showcase a deployed service powered by TextWiser as a proof-of-concept enterprise application.




How to Cite

Kilitcioglu, D., & Kadioglu, S. (2021). Representing the Unification of Text Featurization using a Context-Free Grammar. Proceedings of the AAAI Conference on Artificial Intelligence, 35(17), 15439-15445.



IAAI Technical Track on Innovative Tools for Enabling AI Application