How MySpace data chief makes billions of records add up

Summary:Don Watters, chief data architect at social-networking site MySpace, discusses the problems of dealing with up to six billion visitor records a day going to change completely for us. We are going to need to compare and contrast what happened before and what is going to happen so that we have a good picture of the good parts of the change and what parts might not be hitting with our customers.

How is the use of visitor data evolving?
I've been at MySpace about a year and a half. Before that I was at Disney, ESPN and ABC, and they do a fair amount of traffic as well. They are probably about half the size of MySpace, but they still do a tremendous amount of traffic especially through ESPN. So I do have sense of how things have changed over the years.

What is happening is a big shift in the amount of compute power available and what we can do with it. Having the compute power continue to grow over time is really what is allowing us to do more data analysis and get more value out of the data.

When I first started at Disney we were at probably 3TB total of data in the entire environment. When I left they were probably at about 30TB. Here at MySpace, I'm at one petabyte just in the warehouse and we have something in the order of 2.5 petabytes in production for data development — and that's just for user data. Those numbers will just continue to grow and grow.

So there has been a huge increase in data and computing power, but what about the analysis tools?
We've been working with Aster Data for a while and they have just released a number of analytic functions that allow us to do even more with the data. We've been working with them in terms of getting new functionality into the product, which is extending their SQL-MapReduce framework.

Can you give an example of what these analytic functions allow you to do?
One of the easiest things to understand — but one of the most difficult problems to solve — is a user session. When a user comes on the site they may or may not be logged in, or have a cookie or a consistent IP address from one day to another. We have to try figure out who the user is over the course of the day, so that we can say how long a particular user has spent on the site. We literally look through billions of records a day to try and figure that out.

Part of the issue with that is it's always based on a single user. So most compute solutions would include doing an iterative process to look through those things. However, given that the data is in a database, most database operations are set-based. In other words, they are looking at large amounts of data all at once.

What SQL-MapReduce allows us to do is iterate over that set, in terms of trying to find the user information throughout that set. [This] then gives us the ability to do things such as discover how that user traversed the site in the same session. What categories of the site did they visit — were they part of the music pages or the blog pages or the video pages? — and then show that traversal.

Are the new analytic functions things you have worked on?
SQL-MapReduce has been around a long time. What is happening now is that Aster has come back and got third parties and internal folk from their teams either to build SQL-MapReduce functions or find ones that have been built by other teams, like those from within MySpace. [It has] then coalesced all of those into one package, so that everyone has the opportunity to use them.

What is the most important recent development in data analytics?
Data analytics used to be a very specialised field. It used to be something that you had to have a post-graduate degree to work on, or you had to have very specialised knowledge in a software package that only a few people knew about.

We now have the ability to bring in analysts who know SQL but don't necessarily understand in-depth analysis. Then we only need a few people who understand the specifics behind everything and how to group things together. So I see this as the new age in analysis. We no longer need specialised packages to do in-depth analysis of the data. That change is going to open up a lot of things.

Topics: Networking


Toby Wolpe is a senior reporter at ZDNet in London. He started in technology journalism when the Apple II was state of the art.

Contact Disclosure

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Related Stories

The best of ZDNet, delivered

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
Subscription failed.