Media Cloud: Massive Open Source Collection of Global News on the Open Web

Authors

  • Hal Roberts Berkman Klein Center for Internet & Society, Harvard University Media Cloud
  • Rahul Bhargava College of Arts, Media and Design, Northeastern University Media Cloud
  • Linas Valiukas Media Cloud
  • Dennis Jen Media Cloud
  • Momin M. Malik Berkman Klein Center for Internet & Society, Harvard University
  • Cindy Sherman Bishop Media Cloud
  • Emily B. Ndulue Media Cloud
  • Aashka Dave School of Information & Library Science, University of North Carolina at Chapel Hill
  • Justin Clark Berkman Klein Center for Internet & Society, Harvard University
  • Bruce Etling Berkman Klein Center for Internet & Society, Harvard University Media Cloud
  • Robert Faris Shorenstein Center on Media, Politics and Public Policy, Harvard University Berkman Klein Center for Internet & Society, Harvard University Media Cloud
  • Anushka Shah Civic Studios Media Lab, Massachusetts Institute of Technology
  • Jasmin Rubinovitz Media Lab, Massachusetts Institute of Technology
  • Alexis Hope Media Lab, Massachusetts Institute of Technology
  • Catherine D'Ignazio Department of Urban Studies and Planning, Massachusetts Institute of Technology
  • Fernando Bermejo Media Cloud Berkman Klein Center for Internet & Society, Harvard University
  • Yochai Benkler Harvard Law School, Harvard University Berkman Klein Center for Internet & Society, Harvard University Media Cloud
  • Ethan Zuckerman College of Information & Computer Sciences, University of Massachusetts Amherst Berkman Klein Center for Internet & Society, Harvard University Media Cloud

DOI:

https://doi.org/10.1609/icwsm.v15i1.18127

Keywords:

Human computer interaction; social media tools; navigation and visualization, Analysis of the relationship between social media and mainstream media, Social network analysis; communities identification; expertise and authority discovery, Text categorization; topic recognition; demographic/gender/age identification

Abstract

We present the first full description of Media Cloud, an open source platform based on crawling hyperlink structure in operation for over 10 years, that for many uses will be the best way to collect data for studying the media ecosystem on the open web. We document the key choices behind what data Media Cloud collects and stores, how it processes and organizes these data, and its open API access as well as user-facing tools. We also highlight the strengths and limitations of the Media Cloud collection strategy compared to relevant alternatives. We give an overview two sample datasets generated using Media Cloud and discuss how researchers can use the platform to create their own datasets.
ICWSM-2021 Proceedings Cover

Downloads

Published

2021-05-22

How to Cite

Roberts, H., Bhargava, R., Valiukas, L., Jen, D., Malik, M. M., Bishop, C. S., Ndulue, E. B., Dave, A., Clark, J., Etling, B., Faris, R., Shah, A., Rubinovitz, J., Hope, A., D’Ignazio, C., Bermejo, F., Benkler, Y., & Zuckerman, E. (2021). Media Cloud: Massive Open Source Collection of Global News on the Open Web. Proceedings of the International AAAI Conference on Web and Social Media, 15(1), 1034-1045. https://doi.org/10.1609/icwsm.v15i1.18127

Issue

Vol. 15 (2021): Fifteenth International AAAI Conference on Web and Social Media

Section

Dataset Papers