Yahoo! Pipes tutorial: Build an RSS mashup

I've published a picture gallery that walks through how to use Yahoo! Pipes to mashup several separate RSS feeds into a single 'river of news' feed. It's one of the simplest exercises you can do with Pipes, but as I wrote last Friday, it's a capability that's been on my wishlist for many years.

After the Pipes team fixed the bug that was jinxing my first attempt, I promised to report back when I'd finished building the feed aggregator. What I can tell you is that Pipes does a great job provided your feeds all use the same tag to date each item. The majority of the feeds I was working with use 'pubDate', but I found that I had to eliminate feeds that use other date tags such as 'dc:date' and 'published'. The Pipes team have promised to find a way to normalize dates in feeds as soon as they can. Judging by how quickly they fixed the first bug, I have high hopes that it won't be long before they've cracked this more difficult challenge. Even so, it's great to be able to create this feed so simply. My next task is to publish the updated feed into the original Loosely Coupled news page on which it is based. [Update: date normalization is now fixed and there's also a simple way of publishing a Pipes feed within a web page.]

I admit that it's probably stretching the definition a bit to call this an RSS 'mashup' since all I'm doing is aggregating the feeds and resorting them by date. There are many more interesting combinations that can be done, such as adding Flickr pictures, or related postings from Technorati. But I wanted to describe something that was simple enough for pretty much anyone to follow. Once they've had one successful experience, it's then much easier to go on from there to do more complex stuff.

Besides, it's always an interesting exercise to write a how-to guide for a novice user, because it forces you to think through all the assumed knowledge and context that you would otherwise take for granted. A couple of years ago, I attempted to write a step-by-step account of a test drive of Grand Central's business services network, which was kind of an enterprise-class precursor of Yahoo! Pipes. I failed miserably, at one point wailing:

"... in an ideal world, people like me wouldn't have to go that deep; programmers — who actually enjoy writing raw software code — would create the business-level services for the rest of us, and we'd just assemble them into composite processes without having to worry about arcane stuff like typing, the correct construction of 'for...while' statements and how to evaluate variables."

A couple of years previously I did a similar exercise with a modular system of hosted components that allows website owners to buid forms- and database-driven applications to any website without knowing any HTML. It was less tortuous than the Grand Central test drive but it still showed up the number of steps required to get anything useful done.

The great virtue of Yahoo! Pipes is that even a very simple build can be really useful. One early user describes how he set up a pipe that resorts the feed from his Blogger Beta site:

I have always been frustrated that the new blogger beta sorts its feed by 'updated date' not 'published date'. I make typo corrections to old posts and my readers started complaining that they were seeing my old posts of over a year pop up in their feed.

So I wrote a super simple Yahoo pipe to read my feed, sort it by published date and then that feed can be provided to all my readers.

Once these simple, 'no-brainer' use cases start getting copied, adoption quickly becomes viral, and having got on board people can then start to experiment in more depth. In fact, I think the main challenge for the Yahoo! Pipes team — in addition to overcoming all those niggling mismatches like different RSS formats using different date tags — is going to be managing the discovery process so that users can easily find what other people are doing that works best.