CrisisBench: Benchmarking Crisis-related Social Media Datasets for Humanitarian Information Processing

Authors

  • Firoj Alam QCRI, Doha, Qatar
  • Hassan Sajjad QCRI, Doha, Qatar
  • Muhammad Imran QCRI, Doha, Qatar
  • Ferda Ofli QCRI, Doha, Qatar

DOI:

https://doi.org/10.1609/icwsm.v15i1.18115

Keywords:

Measuring predictability of real world phenomena based on social media, e.g., spanning politics, finance, and health, Qualitative and quantitative studies of social media, Text categorization; topic recognition; demographic/gender/age identification

Abstract

The time-critical analysis of social media streams is important for humanitarian organizations to plan rapid response during disasters. The crisis informatics research community has developed several techniques and systems to process and classify big crisis-related data posted on social media. However, due to the dispersed nature of the datasets used in the literature, it is not possible to compare the results and measure the progress made towards better models for crisis informatics. In this work, we attempt to bridge this gap by combining various existing crisis-related datasets. We consolidate eight annotated data sources and provide 166.1k and 141.5k tweets for informativeness and humanitarian classification tasks, respectively. The consolidation results in a larger dataset that affords the ability to train more sophisticated models. To that end, we provide binary and multiclass classification results using CNN, FastText, and transformer based models to address informativeness and humanitarian tasks, respectively. We make the dataset and scripts available at https://crisisnlp.qcri.org/crisis_datasets_benchmarks.html.

Downloads

Published

2021-05-22

How to Cite

Alam, F., Sajjad, H., Imran, M., & Ofli, F. (2021). CrisisBench: Benchmarking Crisis-related Social Media Datasets for Humanitarian Information Processing. Proceedings of the International AAAI Conference on Web and Social Media, 15(1), 923-932. https://doi.org/10.1609/icwsm.v15i1.18115