Coping With Noise in a Real-World Weblog Crawler and Retrieval System

Authors

  • James Lanagan Clarity: Centre For Sensor Web Technologies
  • Paul Ferguson Clarity: Centre For Sensor Web Technologies
  • Neil O'Hare Clarity: Centre For Sensor Web Technologies
  • Alan Smeaton Clarity: Centre For Sensor Web Technologies

DOI:

https://doi.org/10.1609/icwsm.v4i1.14040

Keywords:

Noise Removal, Blogs, Information Retrieval

Abstract

In this paper we examine the effects of noise when creating a real-world weblog corpus for information retrieval. We focus on the DiffPost (Lee et al. 2008) approach to noise removal from blog pages, examining the difficulties encountered when crawling the blogosphere during the creation of a real-world corpus of blog pages. We introduce and evaluate a number of enhancements to the original DiffPost approach in order to increase the robustness of the algorithm. We then extend DiffPost by looking at the anchor-text to text ratio, and discover that the time-interval between crawls is more important to the successful application of noise-removal algorithms within the blog context, than any additional improvements to the removal algorithm itself.

ICWSM-2010 Cover

Downloads

Published

2010-05-16

How to Cite

Lanagan, J., Ferguson, P., O’Hare, N., & Smeaton, A. (2010). Coping With Noise in a Real-World Weblog Crawler and Retrieval System. Proceedings of the International AAAI Conference on Web and Social Media, 4(1), 271-274. https://doi.org/10.1609/icwsm.v4i1.14040

Issue

Vol. 4 No. 1 (2010): Fourth International AAAI Conference on Weblogs and Social Media

Section

Poster Papers