Limelight Networks: Why the Olympics didn't 'Melt' the Internet

Limelight Networks: Why the Olympics didn't 'Melt' the Internet

Summary: I admit it, even I was skeptical.

SHARE:

I admit it, even I was skeptical.

That was when saw the first demonstration of the Silverlight plugin and the NBCOlympics.com web site back in March of this year at the 2008 Microsoft Technology Summit, where a group of Open Source experts gathered from around the world were asked for feedback on various aspects of Microsoft's emerging technologies.

I truly believed that when they finally rolled that website out, that NBC was going to be overwhelmed with traffic and the site would come to a screeching halt, resulting in a catastrophic embarrassment for Microsoft and NBC.

At the time, the content caching partner for the Olympics that was disclosed to us was Akamai, which is what Microsoft currently uses for hosting its own downloads from MSDN and Microsoft Update.  Akamai uses a centralized data hosting infrastructure with big Internet pipes that mirrors content that is hosted on a customer's own servers.

Usually with the aid of a special caching appliance installed at the customer's ISP or edge network, the request to download that content is re-directed to Akamai's own servers and fat Internet pipes. When you download big ISO CD and DVD images from MSDN, its going right to Akamai's data centers over the public Internet.

As fat as Akamai's pipes are, I've seen MSDN's downloads slow to a crawl during peak download periods, such as the days following Windows XP SP3 and Windows Vista SP1's release. So like my colleagues here at ZDNet, I was expecting the worst.

As it turns out, Akamai is actually used for some, but not all of the cached content used on NBCOlympics.com -- it hosts the "static" content such as the .JPG files and HTML. However, for all the heavy lifting, such as the streaming video, it's all going through infrastructure hosted by Limelight Networks.

Limelight Networks Operations Center

Above: Limelight Networks' Operations Center in Tempe, AZ.

Who are these Limelight guys, anyway? They are a Tempe, Arizona-based company which operates a global network of fiber-optic interconnected datacenters.

Their backbone is capable of 2 Terabits (Tbps)  per second of sustained data transfers and they globally replicate approximately 5 Petabytes  of data on their storage network, which utilize a mix of proprietary vendor SAN replication technology such as EMC SRDF and Open Source-based technology developed by Limelight itself. Surprise! There's some Linux back-ending all that Windows Media.

Back in late July of this year, NBC finally announced that Limelight would be the primary supplier of content caching services for the Olympics. Where Limelight differs from Akamai and why the Internet didn't "melt" is quite simple -- they are completely "off the cloud".  

In other words, unlike Akamai and similar content caching providers, their system isn't deployed over the public Internet.

Also See: Limelight Networks Operations Center Photo Gallery

Say what? Let me explain. When you download videos from NBCOlympics.com, your computer isn't actually going to the Internet to get content. In fact, the content is usually no more than 2 router hops away from your ISP.

Limelight has partnered with over 800 broadband Internet providers worldwide (such as Verizon, Comcast, Road Runner and Optimum Online/Cablevision) so that the content is either co-located in the same facility as your ISP's main communications infrastructure, or it leases a dedicated Optical Carrier line so that it actually appears as part of your ISP's internal network.

In most cases, you're never even leaving your Tier 1 provider to get the video. Slick, isn't it?

limelight-stacktrace.jpg

So how does all that content get there? Live HD video feeds from the Olympics venues are delivered via optical link to NBC's International Broadcast Centre in Beijing.

Next, the Hi-Def signal is then transcoded/downgraded to 480i video resolution using a special CISCO Scientific Atlanta video encoding appliance solution and sent over trans-continental Optical Carrier to NBC Studios in Los Angeles, and then to NBC's broadcast center in 30 Rockefeller Plaza ("30 Rock") in New York City where the encoding to Windows Media Format (WMF) takes place.

Next, via short haul Optical Carrier connection, the Windows Media files are then distributed to Limelight Networks' primary East Coast replicated data center in New York, and then on to the operations center in Tempe, which replicates all 3000+ hours of Olympics video to its global network of ISP co-located data centers and is queued for media streaming at the very edge your local ISP's network (see network trace screen shot above).

To the end user, this is all transparent, and it just plain works.

Localized content caching is going to be the wave of the future, especially when we start seeing lots of "On Demand" content being offered from next-generation media delivery services. If we truly expect stuff like 3G/4G/5G video to be delivered without any hiccups to next generation cell phones and other wireless devices, services like Limelight are going to become increasingly important.

The near-flawless operation of the live video streaming from NBCOlympics.com over the last week is proof that localized content caching technology works.

Will you be implementing localized content caching for your own rich media based sites? Talk Back and let me know.

Topics: Networking, Browser, Data Centers, Hardware, Operating Systems, Software, Storage, Telcos, Windows

About

Jason Perlow, Sr. Technology Editor at ZDNet, is a technologist with over two decades of experience integrating large heterogeneous multi-vendor computing environments in Fortune 500 companies. Jason is currently a Partner Technology Strategist with Microsoft Corp. His expressed views do not necessarily represent those of his employer.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

64 comments
Log in or register to join the discussion
  • It didn't melt but I think it got hot. (nt)

    (nt)
    bjbrock
  • RE: Limelight Networks: Why the Olympics didn't 'Melt' the Internet

    Cali Lewis mentioned her disappointment with the coverage for two reasons on her video podcasts. First, Silverlight does not work on Linux, which is a bummer for the more geeky members of the audience. Second, if any event is popular, meaning ratings and Ad revenue, it is not shown on the web, it is heald for prime time coverage.

    I would like to know if any limelight customers were impacted by all this. I guess it would be hard to find that out...

    Thanks
    zdnetfanuno
  • All those bits flying around makes my head spin...

    I want to subscribe to a video service that has a giant TiVo that will store at least a weeks worth of programing from all the broadcast and cable networks. Then I want them to dish out a program to me on-demand over the net. I could sacrifice quality for convenience. Local and foreign programing would be a nice touch. I think it's doable but all the content providers would have to sign up.
    Mac Hosehead
    • Could be achieved in distributed form

      What I want more than what you describe is the ability for me to stream a "missed" recording from someone else's DVR in the middle of the night to my DVR when it is idle. Or download anything that is on someone else's DVR via a peer to peer content sharing network.

      I use DIRECTV's HD DVR service. They all are also capable of "On Demand" video download. But the On Demand content is very limited. In theory, every DVR from DIRECTV or Cablevison or whatever should be all indexed and connected to a central or distributed database, where any customer can find a "missed" or already recorded program on someone else's machine or groups of machines and download that program to their DVR. Not unlike the way Bittorrent works. The problem is not technical, but licensing and rebroadcast rights and all that stuff. NBC in Los Angeles probably doesnt want people in New York viewing pre-recorded primetime material because it has localized commercials and stuff. And some programs such as sporting events are only be licensed to air for a very limited time, they will need expiration windows for the material if it is shareable. They'll also have to figure out how to have generic content feeds (like what you get in a network downlink at a local affiliate before you broadcast) that are cached and how to inject localized advertisements on the fly. Doable, but not that easy to figure out from a legal point of view.
      jperlow
      • We agree about content providers...

        But in my fantasy world you would not necessarily need a DVR (you could pay extra for a download) and you could get a stream anywhere with net access.
        Mac Hosehead
        • My fantasy is not different from your fantasy

          However, I think what I am proposing is doable with current technology. What you are proposing is doable with technology that will be broadly available within 5 years -- it would need fiber optics to the household. Not unlike what I describe in my Linux in 2016 piece where everything runs on the cloud.
          jperlow
      • it's coming

        Don't worry. I'm sure in the next year or so you will be able to do that. Just not for free.
        Al_nyc
  • RE: Limelight Networks: Why the Olympics didn't 'Melt' the Internet

    Jason,

    Your work is looking better and better! Congrats!

    George Ou
    http://www.ForMortals.com
    georgeou
  • This busts the myth that the Internet works at the same speed for everyone

    This busts the myth that the Internet works at the same speed for everyone. The idea that a small mom & pop shop should be able to deliver video just as fast as a major website is simply a nonstarter. Server capacity and caching capacity must be purchased as with anything of value. This is still a ???level playing field??? because the bandwidth and caching capability is available to anyone at the same price for anyone who could pay for it. A level playing field does not mean every content provider ends up with the same speed as some people would suggest and it???s never been that way.

    Peer-to-peer is a partial exception because it allows anyone to distribute out-of-order data which can't be viewed or heard until after the entire file is finished. That works great for file distribution or video downloads where people don't care about immediate gratification, but it would never work for high quality in-order on demand content.

    There are P2P streaming technologies like TVU but that's limited to the upstream capacity of the typical broadband network and the video streams are very low bit-rate and very low quality. Out-of-order high quality content or in-order low quality content is fine when it???s free (including mostly pirated content), but no commercial content has chosen this free method of distribution. The Olympics were paid for by those Lenovo commercials.
    georgeou
    • So - This Somehow Proves You Were Right Opposing Net Neutrality HOW?

      You're as bad as John McCain, George Ou - you're getting pummelled, but you insist that means you're "winning" for Big Telco, somehow....
      drprodny
      • Do you define neutrality in outcome or neutrality in treatment?

        Do you define neutrality in outcome or neutrality in treatment?

        The deep philosophical debate over Net Neutrality is the meaning of the word ?Neutrality?. Everyone can agree that Neutrality means fairness and equality, but how do we define fairness and equality? Is it about fairness between packets or fairness between people using the network? Is it about ensuring equality in the outcome or ensuring equality in opportunity? Is every website and every consumer entitled to equal performance over the Internet regardless of what they pay, or are they simply entitled to equal opportunity to pay for the same kind of performance? Should one megabit broadband service cost the same as ten megabit broadband service or should everyone be able to buy ten megabit broadband service at the same price?

        Some Net Neutrality advocates argue for an equal outcome in the service that each user or company gets on the Internet. Others like Tim Berners-Lee who is the father of the World Wide Web strongly advocates of Net Neutrality and argues for equal opportunity to pay for the kind of service he desires.

        ?We pay for connection to the Net as though it were a cloud which magically delivers our packets. We may pay for a higher or a lower quality of service. We may pay for a service which has the characteristics of being good for video, or quality audio. But we each pay to connect to the Net, but no one can pay for exclusive access to me.?1

        ?Net Neutrality is NOT asking for the internet for free. Net Neutrality is NOT saying that one shouldn?t pay more money for high quality of service. We always have, and we always will.?2

        1. http://dig.csail.mit.edu/breadcrumbs/node/132
        2. http://dig.csail.mit.edu/breadcrumbs/node/144

        I completely agree with Tim Berners-Lee and his definition of Neutrality and equality being applied to the opportunity to purchase service and not equality in the outcome. However, his position is actually in opposition to many Net Neutrality advocates and many Net Neutrality bills that would ban the sale of QoS and ban exclusive QoS to those who pay.

        If you fundamentally support this type of legislation, then your definition of Neutrality is equality in service regardless of payment or ownership. If that's the case, then you could just as easily argue that LimeLight must equally share its infrastructure to all customers whether they pay a little or a lot.

        Are you prepared to defend this ridiculous notion of equality in outcome instead of the equality in opportunity?

        George Ou
        georgeou
    • Well, not quite

      progressive-download (i.e., watch as you stream) P2P TV is on its way... the P2P-Next project (p2p-next.org) project's Swarm Player, currently in development but available for public trials, is proof that realtime, distributed on-demand video is doable.

      EZTV, a very large collective of people (and large bittorrent tracker) has just started experimenting with the swarm player and 'tstream' files - backwardly-compatible torrent files which also contain the Swarm Player information required to coordinate the pieces into a progressive download... In short, you can (with a little time to let the show start buffering) actually watch files as they stream over bittorrent. I gave it a go with a couple of TV episodes (Daily Show and Colbert Report) and it worked almost flawlessly.


      TVAnts is another example of much smaller-scale bittorrent-based realtime TV streaming, but the P2P-Next's Swarm Player, and the underlying developments and upgrades to the bittorrent technology, finally prove that on-demand, realtime P2P IPTV is actually feasible. In high quality. Available to almost everybody. Of course, the more people who watch and seed after they're done, the more others can enjoy the same content... Share-alike embodied. :)


      The technology is nearly there - given 18-24 months, I'd say it'll still be patiently waiting for the telecommunications industry to catch up! Raw bandwidth to consumers' homes is what we need now, and fewer limits on bandwidth consumption. The ISPs are going to have to rework their business models because this is a real double-edged sword for them, but it's only a good thing for the consumer in the long run.
      ChristopherWoods
  • Great article !! A bit worrying though

    I've always wonder how limelight network competed against Akamai. Nice.

    No wonder Akamai tries so bad to burry that company in law courts with patent violations claims (and apparently is wining most of time).

    Most people don't realize how critical those type of companies are for the internet to survive... What worries me a little is that "completely out of the cloud" mixed with "internet provider parternship" solution doesn't sound really like the Internet spirit. What happens to people that don't have an internet provider which partenered with limelight ? Are they still able to see that video ?
    Obiwan421
  • Video restricted to viewers within the US

    Here is what everybody see when trying to view a video:

    "We're sorry, NBC is required to restrict this video to viewers within the United States"

    Maybe that explain why they didn't 'melt' the Internet?
    lukas4
    • NBC restricted to viewers in the US!

      NBC is the ONLY provider of olympic content in the US as part of the agreement with the IOC. If you are outside the US then NBC is not for you, annother network paid/agreed with the IOC to deliver the Olympics to you.
      johngtimms
  • Locality, locality, locality.

    Yup, yup, yup!!!

    It's all about locality - how big and how close the caches are to the endpoints, and how many/few hops there are in between.

    This is why local computing power will [b]never[/b] go away, and why we will never, ever move towards "pure cloud computing;" it will always be a hybrid. The closer the data is to the CPU, and the less data has to be sent over the network, the faster and more responsive your applications will be. That's just a fact, it will not go away. You will always get better performance if you use the local machine to crunch some of the data.
    CobraA1
  • But how many people really did watch that much?

    I cannot find a single person I know who looked at any of it. Including me.
    dragon@...
    • Well I did!

      I was very happy to be able to watch a few events which were'nt available during my waking hours. It was great being able to watch a few table tennis and badminton matches even though it was after the fact. Give it a try. Even on a slower computer I was able to see the video with little pause occurring
      Cheers
      Bob
      BrewIT
    • I did!

      I did; I have watched several hours of events that didn't make the network broadcast.
      dkerber@...
    • fans of non mainstream sports watched

      I watched quite a bit of the archery coverage. It looks pretty good, but not great. I an see the digital artifacts from the compression process, but it's not much worse than what Time Warner does to some of it's less popular channels. I think it is great because without this I would have maybe seen a 10 minutes of total TV coverage. Now I can see as much of the competition as I want.
      Al_nyc