Building an RSS mashup with Yahoo! Pipes

Next, you'll see a blank canvas. Don't you just hate it when that happens? But don't worry, it won't take long to finish the job.

Notice the list of sources on the left-hand side. You can build pipes using any of these sources.

For your RSS mashup, you'll be using 'Fetch', as shown in the next image.

Click on the 'Fetch' module and drag it onto the canvas.

Fetch is a generic module that can read any RSS feed on the Web. You just need to tell it which ones.

Once you've dragged the module onto the canvas, you can fill in the URLs for the RSS feeds you want in your mashup (you'll need to have the URLs ready to paste in).

You can add any number of URLs to the Fetch module you've created (there is a limit, but it's not documented yet).

To add a URL, simply click on the plus sign next to the word URL at the top, and a new blank field will open at the end of the list. Then type or paste the URL into the space where it says url.

If you make too many blanks, you can delete any spares by clicking on the x to the left of each one.

You'll notice Pipes also added a module called 'Pipe Output' to the canvas. You can ignore this for now. We'll make use of it later on.

This is a good moment to save what you've done so far. This is the Web, after all. You never know when you're going to lose your connection. 'Save early and often' is a good motto.

When you click on the 'Save' button at the top right of the canvas, a blue box appears top left (I've circled it in red on the screenshot). Fill in the name of your mashup and then click on the 'Save' button right next to it.

Now that you've assembled your feeds in the 'Fetch' module, Pipes makes it easy to test and build your mashup. In the menu to the left of the canvas, clicking on the word 'Operators' pulls down a list of modules you'll use to build the mashup.

But first of all, look at the bottom of the canvas and you'll see a gray area labeled 'debugger'. That's where you test your modules. Click on the word 'Refresh' and it'll test the output from whichever module is highlighted in orange. In this case, that's the 'Fetch' module, and as you can see, it works just fine.

This would be a good point to see if you can get someone in the office to ask you what you're doing. Then you can casually reply, "Oh, nothing. I'm just debugging a new software program I've written."

When you move your mouse over the list of operators, a help bubble appears explaining what each one does. You can filter using search terms or other criteria, do a Babel Fish translation into or out of a dozen or so languages, or even use advanced 'For Each' operators to add or change feed items.

We'll be using the 'Sort' operator to rearrange the items in our feed mashup into date order.

Drag the 'Sort' module into an empty space on the canvas.

Now you'll see the pipes concept come to life. You need to connect the 'Fetch' module containing all your feeds to the 'Sort' module.

Click on the small circle at the bottom of the 'Fetch' module and drag it towards the small circle at the top of the 'Sort' module. You'll see a pipe appear and drop into place, as shown in the next picture.

Now that your modules are connected, wait for a moment and Pipes will automagically reconfigure the sort options to match what it finds in your selection of RSS feeds.

For example, notice how the first 'Sort by' option has changed from 'Title' to 'title' with a lower-case 't'. Pipes has replaced its standard option list with the actual field names coming from the connected module.

Clicking on the down arrow shows you the full list of options. In this case it highlights one of the pitfalls of mashing up RSS feeds, because there are several different date tags in use, which I've circled in red.

The most popular RSS formats use a 'pubDate' tag to date feed items. But RSS 1.0 mostly uses 'dc:date' and the up-and-coming Atom format, favored by Blogger, uses 'published'.

Pipes can't sort feeds on all three of these date formats at the same time, although the design team are working on it and hope to have a solution soon.

My workaround to the date format problem is simple but not wholly satisfactory: I've deleted from my 'Fetch' module the two feeds that don't use 'pubDate'. Once the Pipes team come up with a way of normalizing date formats I'll be able to add them back in.

So I've selected 'Sort by' 'pubDate' in 'descending' sort order. This will put the most recent date at the top, creating a "river of news" aggregated feed.

To finish off, connect the 'Sort' module to the 'Pipe Output'. Click on the 'Save' button and you're done.

You can always come back and change things later on. If you hover over a connection, you'll see a pair of scissors appear, which you can click on to sever the connection. Then you can drag in a new module and connect it in place. Popular modules include 'Truncate', which restricts the length of the feed, and 'Unique', which eliminates duplicates.

For now, though, click on 'Back to my pipes' (which I've circled in red) to see your handiwork in action.

Back on the 'My Pipes' page, you can now select your new creation in the list of pipes, and you'll see a screen something like this.

Whenever you want to make changes, click on the schematic or 'View/Edit' to go back to the modules canvas.

To see how your RSS mashup has turned out, click on 'Run this Pipe' (which I've circled in red).

Anytime you 'Run this Pipe' it'll re-run the mashup based on the latest data collected from all of the feeds. This screenshot shows the results from the night before this gallery was published.

But as you can see from the 'Get as RSS' pulldown on the right, you can also feed it into your favorite feedreader or start page. You can even pipe it into any other Yahoo! Pipe that you build.

To see or pipe the latest results, or to view or clone the underlying modules, go to Simple RSS mashup - Loosely Coupled's SOA blog feed (Yahoo! login ID required). I've changed the title, but everything else is exactly as shown in these screenshots.