Introducing Feedcache: QOS for RSS

A short while back I had an itch inside of Skydasher that needed scratching. A big part of the guts of Skydasher deal with handling RSS feeds that users wanted to subscribe to.

Early on we realized that displaying RSS feeds in a web browser was a lot harder than just grabbing an XML file and parsing out the results. For instance – there can be a lot of overhead to the transaction – overhead that might be more appropriately dealt with asynchronously. Then there’s the question of formats. Not every feed in the universe is published using the de facto RSS 2.0 standard. Some are published using older versions of the RSS specification, and some exist in alternative formats like ATOM. Then there’s the question of feed reliability. The net isn’t a terribly stable place. The number of error conditions that can be encountered when trying to retrieve a feed are almost endless.

This all adds up to a lot of complexity for developers that are just looking to retrieve a feed and display it in an application. We realized that we would encounter this complexity every time that we added RSS support to any of our applications. Looking at all of this, we realized that we needed to build some widget-grease that gave us a “build once, deploy often” capability.

So we built a feed caching engine and stuck an API on it that we made available to any internal applications that needed to suck in RSS feeds and display them in a web page. The way it works is remarkably simple – we send a request for a feed to the API and the API responds with a feed. If the cache doesn’t have the feed in its index, it goes out and gets it so that the next time the feed is requested, the RSS is available. This is a simplified view of the tool, but it also does regular harvesting, rationalizes normalizes alternative feed formats into RSS 2.0, stores binary objects and so on. Of course, if my description is lacking, you could also check it out for yourself.

We recently realized that Feedcache might be useful for other developers as well, so we’ve created a slightly modified version of Feedcache specifically for developers to hack around with. Officially (well, unofficially, this isn’t a product that we’re selling, its a program we’re offering) called “Tucows Feedcache: Developer Edition Beta” (heh – beat that Microsoft!) that employs developer keys to track who’s who (to track and prevent abuse) and give access to everyone who wants it.

You can get more details about Feedcache over at the Feedcache developer wiki. My personal hope is that developers find this useful and help us improve those bits that they don’t yet find useful. Let me know what you think.