X

Building an RSS mashup with Yahoo! Pipes

Yahoo! Pipes makes it really easy to build a mashup of your favorite RSS feeds, even if you're clueless about programming. Anyone can follow this step-by-step guide to see how it's done.
By Phil Wainewright, Contributor
54749.jpg
1 of 14 Phil Wainewright/ZDNET
Yahoo! Pipes makes it really easy to build a mashup of your favorite RSS feeds, even if you're clueless about programming. Anyone can follow this step-by-step guide to see how it's done.

First of all, log in to Yahoo! Pipes and go to the 'my pipes' page.

Click on 'Create a New Pipe' (I've circled it in red on the screenshot).
54750.jpg
2 of 14 Phil Wainewright/ZDNET
Next, you'll see a blank canvas. Don't you just hate it when that happens? But don't worry, it won't take long to finish the job.

Notice the list of sources on the left-hand side. You can build pipes using any of these sources.

For your RSS mashup, you'll be using 'Fetch', as shown in the next image.
54751.jpg
3 of 14 Phil Wainewright/ZDNET
Click on the 'Fetch' module and drag it onto the canvas.

Fetch is a generic module that can read any RSS feed on the Web. You just need to tell it which ones.

Once you've dragged the module onto the canvas, you can fill in the URLs for the RSS feeds you want in your mashup (you'll need to have the URLs ready to paste in).
54752.jpg
4 of 14 Phil Wainewright/ZDNET
You can add any number of URLs to the Fetch module you've created (there is a limit, but it's not documented yet).

To add a URL, simply click on the plus sign next to the word URL at the top, and a new blank field will open at the end of the list. Then type or paste the URL into the space where it says url.

If you make too many blanks, you can delete any spares by clicking on the x to the left of each one.

You'll notice Pipes also added a module called 'Pipe Output' to the canvas. You can ignore this for now. We'll make use of it later on.
54753.jpg
5 of 14 Phil Wainewright/ZDNET
This is a good moment to save what you've done so far. This is the Web, after all. You never know when you're going to lose your connection. 'Save early and often' is a good motto.

When you click on the 'Save' button at the top right of the canvas, a blue box appears top left (I've circled it in red on the screenshot). Fill in the name of your mashup and then click on the 'Save' button right next to it.
54754.jpg
6 of 14 Phil Wainewright/ZDNET
Now that you've assembled your feeds in the 'Fetch' module, Pipes makes it easy to test and build your mashup. In the menu to the left of the canvas, clicking on the word 'Operators' pulls down a list of modules you'll use to build the mashup.

But first of all, look at the bottom of the canvas and you'll see a gray area labeled 'debugger'. That's where you test your modules. Click on the word 'Refresh' and it'll test the output from whichever module is highlighted in orange. In this case, that's the 'Fetch' module, and as you can see, it works just fine.

This would be a good point to see if you can get someone in the office to ask you what you're doing. Then you can casually reply, "Oh, nothing. I'm just debugging a new software program I've written."
54755.jpg
7 of 14 Phil Wainewright/ZDNET
When you move your mouse over the list of operators, a help bubble appears explaining what each one does. You can filter using search terms or other criteria, do a Babel Fish translation into or out of a dozen or so languages, or even use advanced 'For Each' operators to add or change feed items.

We'll be using the 'Sort' operator to rearrange the items in our feed mashup into date order.

Drag the 'Sort' module into an empty space on the canvas.
54756.jpg
8 of 14 Phil Wainewright/ZDNET
Now you'll see the pipes concept come to life. You need to connect the 'Fetch' module containing all your feeds to the 'Sort' module.

Click on the small circle at the bottom of the 'Fetch' module and drag it towards the small circle at the top of the 'Sort' module. You'll see a pipe appear and drop into place, as shown in the next picture.
54757.jpg
9 of 14 Phil Wainewright/ZDNET
Now that your modules are connected, wait for a moment and Pipes will automagically reconfigure the sort options to match what it finds in your selection of RSS feeds.

For example, notice how the first 'Sort by' option has changed from 'Title' to 'title' with a lower-case 't'. Pipes has replaced its standard option list with the actual field names coming from the connected module.
54758.jpg
10 of 14 Phil Wainewright/ZDNET
Clicking on the down arrow shows you the full list of options. In this case it highlights one of the pitfalls of mashing up RSS feeds, because there are several different date tags in use, which I've circled in red.

The most popular RSS formats use a 'pubDate' tag to date feed items. But RSS 1.0 mostly uses 'dc:date' and the up-and-coming Atom format, favored by Blogger, uses 'published'.

Pipes can't sort feeds on all three of these date formats at the same time, although the design team are working on it and hope to have a solution soon.
54759.jpg
11 of 14 Phil Wainewright/ZDNET
My workaround to the date format problem is simple but not wholly satisfactory: I've deleted from my 'Fetch' module the two feeds that don't use 'pubDate'. Once the Pipes team come up with a way of normalizing date formats I'll be able to add them back in.

So I've selected 'Sort by' 'pubDate' in 'descending' sort order. This will put the most recent date at the top, creating a "river of news" aggregated feed.
54760.jpg
12 of 14 Phil Wainewright/ZDNET
To finish off, connect the 'Sort' module to the 'Pipe Output'. Click on the 'Save' button and you're done.

You can always come back and change things later on. If you hover over a connection, you'll see a pair of scissors appear, which you can click on to sever the connection. Then you can drag in a new module and connect it in place. Popular modules include 'Truncate', which restricts the length of the feed, and 'Unique', which eliminates duplicates.

For now, though, click on 'Back to my pipes' (which I've circled in red) to see your handiwork in action.
54761.jpg
13 of 14 Phil Wainewright/ZDNET
Back on the 'My Pipes' page, you can now select your new creation in the list of pipes, and you'll see a screen something like this.

Whenever you want to make changes, click on the schematic or 'View/Edit' to go back to the modules canvas.

To see how your RSS mashup has turned out, click on 'Run this Pipe' (which I've circled in red).
54762.jpg
14 of 14 Phil Wainewright/ZDNET
Anytime you 'Run this Pipe' it'll re-run the mashup based on the latest data collected from all of the feeds. This screenshot shows the results from the night before this gallery was published.

But as you can see from the 'Get as RSS' pulldown on the right, you can also feed it into your favorite feedreader or start page. You can even pipe it into any other Yahoo! Pipe that you build.

To see or pipe the latest results, or to view or clone the underlying modules, go to Simple RSS mashup - Loosely Coupled's SOA blog feed (Yahoo! login ID required). I've changed the title, but everything else is exactly as shown in these screenshots.

Related Galleries

Holiday wallpaper for your phone: Christmas, Hanukkah, New Year's, and winter scenes
Holiday lights in Central Park background

Related Galleries

Holiday wallpaper for your phone: Christmas, Hanukkah, New Year's, and winter scenes

21 Photos
Winter backgrounds for your next virtual meeting
Wooden lodge in pine forest with heavy snow reflection on Lake O'hara at Yoho national park

Related Galleries

Winter backgrounds for your next virtual meeting

21 Photos
Holiday backgrounds for Zoom: Christmas cheer, New Year's Eve, Hanukkah and winter scenes
3D Rendering Christmas interior

Related Galleries

Holiday backgrounds for Zoom: Christmas cheer, New Year's Eve, Hanukkah and winter scenes

21 Photos
Hyundai Ioniq 5 and Kia EV6: Electric vehicle extravaganza
img-8825

Related Galleries

Hyundai Ioniq 5 and Kia EV6: Electric vehicle extravaganza

26 Photos
A weekend with Google's Chrome OS Flex
img-9792-2

Related Galleries

A weekend with Google's Chrome OS Flex

22 Photos
Cybersecurity flaws, customer experiences, smartphone losses, and more: ZDNet's research roundup
shutterstock-1024665187.jpg

Related Galleries

Cybersecurity flaws, customer experiences, smartphone losses, and more: ZDNet's research roundup

8 Photos
Inside a fake $20 '16TB external M.2 SSD'
Full of promises!

Related Galleries

Inside a fake $20 '16TB external M.2 SSD'

8 Photos