Empirical Evaluation of Three Common Assumptions in Building Political Media Bias Datasets

Authors

  • Soumen Ganguly Saarland Informatics Campus
  • Juhi Kulshrestha GESIS - Leibniz Institute for the Social Sciences
  • Jisun An Qatar Computing Research Institute, HBKU
  • Haewoon Kwak Qatar Computing Research Institute, HBKU

Abstract

In this work, we empirically validate three common assumptions in building political media bias datasets, which are (i) labelers' political leanings do not affect labeling tasks, (ii) news articles follow their source outlet's political leaning, and (iii) political leaning of a news outlet is stable across different topics. We build a ground-truth dataset of manually annotated article-level political leaning and validate the three assumptions. Our findings warn that the three assumptions could be invalid even for a small dataset. We hope that our work calls attention to the (in)validity of common assumptions in building political media bias datasets.

Downloads

Published

2020-05-26

How to Cite

Ganguly, S., Kulshrestha, J., An, J., & Kwak, H. (2020). Empirical Evaluation of Three Common Assumptions in Building Political Media Bias Datasets. Proceedings of the International AAAI Conference on Web and Social Media, 14(1), 939-943. Retrieved from https://ojs.aaai.org/index.php/ICWSM/article/view/7362