Construction of Evaluation Datasets for Trend Forecasting Studies


  • Shogo Matsuno Gunma University
  • Sakae Mizuki Hottto Link, inc.
  • Takeshi Sakaki Hottto Link, inc.



, Trend identification and tracking; time series forecasting


In this study, we discuss issues in the traditional evaluation norms of trend forecasts, outline a suitable evaluation method, propose an evaluation dataset construction procedure, and publish Trend Dataset: the dataset we have created. As trend predictions often yield economic benefits, trend forecasting studies have been widely conducted. However, a consistent and systematic evaluation protocol has yet to be adopted. We consider that the desired evaluation method would address the performance of predicting which entity will trend, when a trend occurs, and how much it will trend based on a reliable indicator of the general public's recognition as a gold standard. Accordingly, we propose a dataset construction method that includes annotations for trending status (trending or non-trending), degree of trending (how well it is recognized), and the trend period corresponding to a surge in recognition rate. The proposed method uses questionnaire-based recognition rates interpolated using Internet search volume, enabling trend period annotation on a weekly timescale. The main novelty is that we survey when the respondents recognize the entities that are highly likely to have trended and those that haven't. This procedure enables a balanced collection of both trending and non-trending entities. We constructed the dataset and verified its quality. We confirmed that the interests of entities estimated using Wikipedia information enables the efficient collection of trending entities a priori. We also confirmed that the Internet search volume agrees with public recognition rate among trending entities.




How to Cite

Matsuno, S., Mizuki, S., & Sakaki, T. (2023). Construction of Evaluation Datasets for Trend Forecasting Studies. Proceedings of the International AAAI Conference on Web and Social Media, 17(1), 1041-1051.