Measuring Societal Biases from Text Corpora with Smoothed First-Order Co-occurrence


  • Navid Rekabsaz Johannes Kepler University, Linz Institute of Technology, AI Lab
  • Robert West EPFL
  • James Henderson Idiap Research Institute
  • Allan Hanbury TU Wien, Complexity Science Hub Vienna



Subjectivity in textual data; sentiment analysis; polarity/opinion identification and extraction, linguistic analyses of social media behavior, Text categorization; topic recognition; demographic/gender/age identification


Text corpora are widely used resources for measuring societal biases and stereotypes. The common approach to measuring such biases using a corpus is by calculating the similarities between the embedding vector of a word (like nurse) and the vectors of the representative words of the concepts of interest (such as genders). In this study, we show that, depending on what one aims to quantify as bias, this commonly-used approach can introduce non-relevant concepts into bias measurement. We propose an alternative approach to bias measurement utilizing the smoothed first-order co-occurrence relations between the word and the representative concept words, which we derive by reconstructing the co-occurrence estimates inherent in word embedding models. We compare these approaches by conducting several experiments on the scenario of measuring gender bias of occupational words, according to an English Wikipedia corpus. Our experiments show higher correlations of the measured gender bias with the actual gender bias statistics of the U.S. job market – on two collections and with a variety of word embedding models – using the first-order approach in comparison with the vector similarity-based approaches. The first-order approach also suggests a more severe bias towards female in a few specific occupations than the other approaches.




How to Cite

Rekabsaz, N., West, R., Henderson, J., & Hanbury, A. (2021). Measuring Societal Biases from Text Corpora with Smoothed First-Order Co-occurrence. Proceedings of the International AAAI Conference on Web and Social Media, 15(1), 549-560.