Is RSS Clogging The Web?

Wired.com asks "Will RSS Readers Clog the Web"?

The article raises some interesting points. As more people take advantage of feeds, the syndicating sites have to dedicate more bandwidth to serving them.

However, many developers of newsreaders are not complying with HTTP specifications and are requesting the full feed file each time instead of checking the files modification time. Other readers are checking modification time against the local machines clock instead of the webserver which causes problems if they are not in sync with each other.

I made a quick check of my access logs to see what was happening with my RSS feeds. Two sites poll once an hour, and several others on a less frequent basis.

Of the two hourly pollers, Bloglines was behaving itself and sends an If-Modified-Since header meaning my webserver doesn't send any content, just a 304 Not Modified status code if there isn't any new content.

However the second site, The UK Weblogs Aggregator, doesn't and requests a full feed each time.

I should make some changes to my RSS feed files. At present my RSS 1.0 Feed does not use the RSS 1.0 syndication module. If I added this information to the feed then I can state exactly how often the feed is updated or how often I want aggregators to check it. The RSS 0.91 specification gives users the option of saying what hours and days the feed should not be accessed with the <skipDay> and <skipHours> tags, and this also applies to compatible RSS 2.0 format. Atom is the other popular syndication format, however this doesn't seem to have a formal method for controlling access, so relies on HTTP.

The feeds on RobertPrice.co.uk are cut down, and only carry the first few hundred characters of the five most recent entries, along with some supporting data. However, some sites include the full body of their content, and aggregators behaving badly can waste bandwidth and server capacity for those sites.