An Optimized Web Feed Aggregation Approach for Generic Feed Types

Abstract

Web feeds are a popular way to access updates for contentin the World Wide Web. Unfortunately, the technology be-hind web feeds is based on polling. Thus, clients ask the feedserver regularly for updates. There are two concurrent prob-lems with this approach. First, many times a client asks forupdates, there is no new item and second, if the client’s up-date interval is too large it might be notified too late or evenmiss items. In this work we present adaptive feed polling algorithms. Thealgorithms learn from the previous behaviors of feeds andpredict their future behaviors. To evaluate these algorithmswe created a real set of over 180,000 diversified feeds andcollected a dataset of their updates for a time of three weeks.We tested our adaptive algorithms on this set and show thatadaptive feed polling reduces traffic significantly and pro-vides near-real-time updates.