A rational debate on Comcast traffic management

A rational debate on Comcast traffic management

Summary: Updated 8:00 PMThe discussion on Comcast actively resetting BitTorrent connections to manage its network for its cable broadband service has gotten hot in recent weeks and there hasn't been a whole lot of accurate reporting on the subject because of the complexity of the issue. The subject of Net Neutrality has once again surfaced with Comcast's actions being the latest rallying cry of Internet "discrimination".

SHARE:

Updated 8:00 PM The discussion on Comcast actively resetting BitTorrent connections to manage its network for its cable broadband service has gotten hot in recent weeks and there hasn't been a whole lot of accurate reporting on the subject because of the complexity of the issue. The subject of Net Neutrality has once again surfaced with Comcast's actions being the latest rallying cry of Internet "discrimination". This isn't the first time an ISP was used as an example of Internet discrimination, extremists concocted a story that Cox was blocking Craigslist when the problem was arguably Craigslist own making all along.

Much like the debate on Net Neutrality, I sat on the sidelines for a while to get a handle on the situation. I had mixed feelings on the issue and I didn't jump in until I saw the debate get out of hand with outright nonsense. Since June of 2006 I've written a series of articles on the issue of Net Neutrality culminating in "A rational debate on Net Neutrality" which I can proudly say got a lot of linking from reasonable proponents on both sides of the issue. Now that extremist groups like SaveTheInternet.com is lobbying the FCC to stop Comcast's network management practices without fully understanding what they are asking for, I'm going to try and start a rational debate on the issue.

I will start by summarizing the Comcast situation and how it all started. Comcast was found to be actively resetting TCP connections on BitTorrent peer-to-peer file trading connections by forging TCP reset packets that appear to be coming from the BitTorrent peers. When most of us hear the term "forged TCP reset packets", it sounds like Comcast has crossed the line of reasonable network management Comcast is guilty of application discrimination. So when word of this got out, all hell broke loose and the knifes were out for Comcast's blood.

The Free Press has gone as far as demanding an FCC enjoinment before the merits are even decided and they are demanding fines of $195,000 per infraction which would amount to $2.3 TRILLION dollars if we only counted Comcast customers.

The FCC should act immediately to enjoin Comcast’s secret discrimination and, even before deciding the merits, issue a temporary injunction requiring Comcast to stop degrading any applications. Upon deciding the merits, the Commission should issue a permanent injunction ending Comcast’s discrimination. The Commission should also impose the maximum forfeitures to deter Comcast and other network providers and to ensure society is fully compensated for the harms imposed by Internet discrimination.

I have to admit that when I first heard about the issue, I too thought Comcast crossed the line of reasonable network management in to abusive behavior but after speaking to Richard Bennett who had a hand in creating some of the technology used to build the Internet, I'm not so certain that my initial assessment was correct.

<Next page - Web hogs plus shared cable equals network meltdown>

Web hogs plus shared cable equals network meltdown

I consider myself somewhat knowledgeable on networking technology since I was an IT consultant who designed and built switching and routing infrastructure for clients, but I'm certainly no match for Richard Bennett. So when I argued with Richard on why Comcast had gone too far, Richard explained that my arguments didn't apply to a shared medium network. The type of networks that I designed and deployed in the last 6 years were all switched and routed networks and I had the luxury of dedicated 100-megabit or gigabit interconnects between each user and the switch. With switches being ubiquitous within the last 6 years, I had almost forgotten about the bad old days of hub topology where every device shared a single physical connection with a single collision domain where data collided if too many end points transmitted data at once.

There is no advanced queuing capability on a shared cable medium (contention traffic is dropped, not queued) and whenever there was excessive traffic, you had a situation of degraded aggregate performance and everyone suffered.  If the traffic went past a certain point of critical mass, the problems amplified and we use to call those broadcast storms where the network basically melted down and you pretty much couldn't transmit anything.  Comcast is a cable broadband company and cable broadband is fundamentally a shared medium network where cable customers in the same neighborhood share a single network connection.  As a matter of fact, SBC (currently AT&T) use to lambaste their Cable broadband competitors with the jargon "Web hog" in the following hilarious video commercial that most people probably remember.

Since this is obviously something Comcast would rather not talk about, it probably explains why Comcast hasn't been more forthcoming. I should note that Cable isn't nearly as bad as the kind of experiences I've had with AT&T DSL service.

Comcast doesn't actually block BitTorrent usage and several people I know have no issues getting BitTorrent to work with legal or illegal copyrighted content. What Comcast does is actively reduce the dozens of simultaneous BitTorrent upload connections that a user can have. As Richard Bennett explained it to me, this is content- and viewpoint-neutral and it isn't "content-based discrimination" as so many make it out to be.  In fact the more I examine this issue, the more it looks like reasonable network management to me.

We can think of it as a freeway onramp that has lights on it to rate limit the number of cars that may enter a freeway.  Those lights aren't there to say people of a certain race can pass through or people of a certain race must wait longer in line; everyone must wait their turn.  If you didn't have the lights and everyone tries to pile on to the freeway at the same time, everyone ends up with worse traffic.  Comcast doesn't block you from using BitTorrent, it simply limits the number of simultaneous uploads you can perform at once.

<Next page - How cable modems work>

How cable modems work

Richard Bennett explained to me that cable modems have to send an RTS (Request To Send) anytime it wants to transmit data.  When two or more cable modems happen to send an RTS at the same time and collide, the RTS is dropped and then no one gets to transmit anything.  The problem actually compounds since the more collisions there are the more send requests pile up on the cable modems throughout the network which increases the chances of even more RTS collisions.  The only way to stop this problem get the users to reduce the number of transmissions and there simply is no other mechanism that can manage this type of a network other than forcing overly aggressive clients to reduce the number of simultaneous connections with forged TCP reset packets.

Richard Bennett: Cable modems have a crappy upstream protocol. When it wants to send, it sends a request to send packet to the controller, and waits for a reply that gives it a time slot. But the RTS packet is sent in a contention slot, such that any two stations sending RTS in the same cycle will collide, and then nobody gets to transmit. The more data you have queued at the cable modem, the more likely a collision.

The network is physically large, with a long propagation delay relative to the size of the collision window. And when collisions start to happen, they ripple as more and more stations have data queued for transmission. So the only way to make this protocol stable is to actively limit the amount of data queued at the cable modem for upstream delivery, and only way to do that for Torrent is to stifle connections at the TCP level. I've tried to scheme up a better way to do this, and there isn't one.

Simply put, there is no queue for you to prioritize in the first place on a cable broadand network.This isn't the prettiest solution in the world but there is nothing pretty about a shared collision domain network topology and there aren't any other solutions other than active network management.  Conventional QoS (Quality of Service) priority queuing works on a router which comprises most of the Internet but it has no effect on a shared last-mile collision domain network where packets are simply discarded if they collide.  Simply put, there is no queue for you to prioritize in the first place.  Actively managing the number of simultaneous uploads cable broadband BitTorrent users improves performance for everyone and every application including BitTorrent.  In fact a well known trick among advanced BitTorrent users is to limit the number of simultaneous connections to optimize and increase performance but Comcast is doing this at the cable network level to optimize and increase everyone's performance.

The Free Press is demanding that Comcast implement mechanisms that simply don't exist.

Free Press: More importantly, if Comcast is concerned that the collective set of users running P2P applications are affecting quality of service for other users on a cable loop, they could readily set dynamic quotas for each user on the loop, so as to ensure that there is always bandwidth available for users who are not running P2P applications – and they could do so without interfering in protocol choice. Or they could also charge by usage, provide more bandwidth to all users, or actually offer high symmetric broadband speeds.

The fact that Free Press would suggest that network carriers "charge by usage" (metered Internet) goes against every principle of the InternetOf course it would be nice if there were a dynamic network management protocol built in to the cable modems that actively manage traffic without the use of ugly TCP reset to manage excessive traffic, but such a mechanism doesn't exist and the Free Press is being ignorant about reality.  Not only are they reckless for demanding the FCC shut down the current traffic management system, their proposed solutions simply have no effect on the RTS collision problem and their solution harms the consumer.  The fact that Free Press would suggest that network carriers "charge by usage" goes against every principle of the Internet and it would actually put a big fat smile on network carrier executives.  I can't believe that true consumer advocate groups haven't picked up on this since it would greatly harm the Internet and the consumer.

BitTorrent is by far the largest consumer of bandwidth and a single BitTorrent user is capable of generating hundreds of times more network load than conventional applications.  Throttling the number of BitTorrent connections or any application that has similarly aggressive characteristics is critical to keeping the network healthy with reasonable round-trip response times.  That means a better gaming and VoIP (Voice over Internet Protocol) experience since they are both highly sensitive to network latency despite the fact that they are low-bandwidth.  If the Net Neutrality extremists get their way and get the Government to ban active network management, cable broadband customers will suffer and those web hog TV commercials might just come true.

Update 11/8/2007 - Speaking of rational debates, here's a great discussion on this Comcast issue.

<Return to top>

Topics: Mobility, Hardware, Networking, Telcos

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

220 comments
Log in or register to join the discussion
  • Excellent explananation!

    Your explanation and rationale make sense. Although I've had no
    speed problems with my Comcast cable internet service (it seems
    to run as fast as my employer's corporate lan) I am left a little
    disappointed with the less than lightning fast page loads that the
    "advertising hype" seem to imply I should be getting.
    Maybe the marketing types are setting our expectations too high
    in order to sell their product?
    kd5auq
    • What! Marketing? Expectations? No, never :D

      What! Marketing? Expectations? No, never ;).

      Thanks for your comments.
      georgeou
    • Page Downloading Issues

      Slow page downloads aren't necessarily Comcast's fault. The 6 MBps they promise you is probably still there for transferring data between Comcast's routers and your PC, but much of the delay you are experiencing can be attributed to the server and to routing delays on the Internet itself. If you are downloading a website that consists of lots of small resources, it might not load much faster on Comcast (or even FIOS, for that matter) than with dialup.
      psomerset@...
  • The legality of Comcast's chosen method is questionable

    In Connecticut for example, it is simply illegal for anyone to electronically pretend to be someone else. By forging RST packets, Comcast is pretending to be someone else.


    "Comcast doesn?t block you from using BitTorrent, it simply limits the number of simultaneous uploads you can perform at once."

    In my experience, that number is very often set to zero. That is, Comcast makes it difficult to impossible to seed a torrent. While technically true that Comcast doesn't prevent you from using BitTorrent, that's rather like like saying that blocking HTTP doesn't prevent you from using a Web browser.


    "If the Net Neutrality extremists get their way and get the Government to ban active network management, cable broadband customers will suffer and those web hog TV commercials might just come true."

    Or Comcast can take some of their profits and run them back into improving their network to handle the traffic.



    I appreciate your explanation of why Comcast's network is FUBAR'ed. It's very similar to how Boston's Central Artery used to be. Boston eventually had to scrap the Central Artery. Perhaps Comcast needs to scrap their current architecture.
    Letophoro
    • Every Comcast customer I've talked to contracts your experience

      "In Connecticut for example, it is simply illegal for anyone to electronically pretend to be someone else. By forging RST packets, Comcast is pretending to be someone else."

      You're a lawyer and judge of this? This is network management below the application layer. This is done on private networks all the time to manage their network so are they breaking the law too? Comcast does not alter contents; they're altering network management packets to make the network perform better for EVERYONE including BitTorrent users.

      Every ISP have it in their contract that if you generate so much load that you affect other customers, they can block you from doing so. Even FiOS has 8 users on a shared fiber run and they have that in the contract. Comcast has it in their contract that they have the right to manage traffic.

      "In my experience, that number is very often set to zero. That is, Comcast makes it difficult to impossible to seed a torrent. While technically true that Comcast doesn't prevent you from using BitTorrent, that's rather like saying that blocking HTTP doesn't prevent you from using a Web browser."

      Every Comcast customer I've talked to contracts your experience. They've had no problems downloading, uploading, or seeding if it isn't excessive. If you limited your upload bandwidth usage and number of connections, it shouldn't be a problem. If enough Comcast customers became seeds at full throttle, it would make the network suffer.

      "Or Comcast can take some of their profits and run them back into improving their network to handle the traffic."

      Everyone would like faster speeds. I would LOVE it if our freeways were 10+ lanes wide but they aren't so we have to have traffic lights that limit the rate of merging traffic to make the roads more usable. What Comcast is doing is making the most out of the existing network. They're working on upgrading their network but that would only mitigate the need for network management and not eliminate it.
      georgeou
      • Don't get bent out of shape.

        "You're a lawyer and judge of this?"
        IANAL - Neither are you, but here's a [url=http://www.jud.state.ct.us/CriminalJury/11-13.html]link[/url] to the applicable statute. Since Comcast clearly derives a benefit(staving off infrastructure improvements and saving on bandwidth) by harming or defrauding others (interfering with their file transfers), they may be criminally liable.

        "This is done on private networks all the time to manage their network so are they breaking the law too?"
        What packets are forged to cause disconnects on private networks?

        "Comcast does not alter contents; they're altering network management packets to make the network perform better for EVERYONE including BitTorrent users."
        How does interfering with BitTorrent transfers improve performance for those using BitTorrent? And if you could, explain how forged RST packets are altered network management packets.

        "Every ISP have it in their contract that if you generate so much load that you affect other customers, they can block you from doing so."
        That's a slippery clause at best. Any traffic can affect others.

        "Every Comcast customer I've talked to contracts your experience. They've had no problems downloading, uploading, or seeding if it isn't excessive. If you limited your upload bandwidth usage and number of connections, it shouldn't be a problem. If enough Comcast customers became seeds at full throttle, it would make the network suffer."
        I can only assume that you meant 'contradicts.' Given that you probably have talked to customers in only a few areas, you can hardly be fully informed as to the problem elsewhere. Comcast has not fully implemented this packet forgery everywhere. As to limiting my bandwidth usage and connections - I DO. If there a 40 in my swarm and I am unable to maintain even one connection today when months ago I could maintain several, I have to believe that Comcast is futzing with my transfers.

        "What Comcast is doing is making the most out of the existing network. They're working on upgrading their network but that would only mitigate the need for network management and not eliminate it."
        My observation is that Comcast appears to trying to get more users onto their current network. They may be trying to improve it, but forging packets is simply not network management.
        Letophoro
        • Forging packets is done all the time for network management

          Forging packets is done all the time for network management. It would be nice if there were a dynamic network management client that can be centrally controlled built in to the cable modems but that does not exist. The FreePress formal complaint demands that the FCC act before the merits of the complaint are even examined. They demand that comcast use a dynamic management protocol when no such mechanism exists. They are in essense saying we don't like the current traffic light management system and we want you to take down the current traffic light system now.
          georgeou
          • I know this bunch

            There would be a speed problem, so Comcast plays traffic cop. Then because they get caught playing traffic cop, they get busted. So they stop playing traffic cop and everyone complains because the network is too slow. You can't win for losing.
            nucrash
          • List some examples if you don't mind.

            Keep in mind that the examples must clearly show that the packet is forged so as to appear to originate from other than the actual origin. NAT translations don't count.


            "They are in essense saying we don't like the current traffic light management system and we want you to take down the current traffic light system now."

            Traffic lights or snipers shooting out your tires? If it were simply traffic lights, then all traffic would be affected. Instead Comcast has taken the route of targeting one specific class of traffic. Perhaps instead of traffic lights you could more accurately describe it as selectively thinning the herd.
            Letophoro
          • What do you mean NAT doesn't count?

            You're are forging packets to make them appear as if they're coming from somewhere else. Are you going to arrest someone or ask for a trillion dollar fine and demand they stop using NAT?

            "Instead Comcast has taken the route of targeting one specific class of traffic. Perhaps instead of traffic lights you could more accurately describe it as selectively thinning the herd. "

            Actually it doesn't target any specific content nor does it stop you from using BitTorrent. It simply prevents excessive BitTorrent upload connections that bring the entire network down to a crawl if a large enough minority do it. Comcast has the right to manage the network and the formal complaint from Free Press doesn't dispute that. But what Free Press wants is a pie-in-sky solution that simply does not exist today and they want a pie-in-sky trillion dollar fine.
            georgeou
          • Why NAT doesn't count.

            It's really very simple. NAT is used to actually route traffic and does not pretend that the origin of the packet is other than the origin. Or to put it simply, you can reply to a NAT'ed address and actually get back to the origin. Alternatively, you can think of NATing as taking a bunch of your outbound letters and putting them in a single envelope with a return address of your mailroom. A reply to your mailroom will still get back to you.

            Again: Keep in mind that the examples must clearly show that the packet is forged so as to appear to originate from other than the actual origin.


            "Actually it doesn't target any specific content nor does it stop you from using BitTorrent."
            Agreed, it doesn't target specific content. Rather like banning all products that consist of ink printed on paper does not specifically target pornography or Bibles. As for stopping the use of BitTorrent, I reiterate that it's rather like like saying that blocking HTTP does not prevent you from using a Web browser. While technically true, it is disingenuous.


            Now, as you are the network expert and I am not... I again request that you cite some examples of forged packet used for network management on private networks.
            Letophoro
          • Actually, NAT can change the origin and destination

            "It's really very simple. NAT is used to actually route traffic and does not pretend that the origin of the packet is other than the origin."

            Actually, NAT can change the origin and destination IP addresses and makes it look like something else.
            georgeou
          • NAT does not forge packets. Now cite your examples and stop dissembling.

            NAT does not forge packets. It readdresses or repackages them. Much like a mailroom for a corporation or the USPS may readdress or repackage mail. Since there is no attempt to defraud anyone, NAT would not be and is not illegal. Or to be more clear, replying to the address in the from field of a NAT'ed packet will actually find the originator of said packet.

            Comcast forges packets. It creates packets that did not exist for the sole purpose of interrupting communications between the original addressees. These packets are indistinguishable from legitimate packets. There is an obvious attempt to harm others for a gain.

            [b]Now, as you are the network expert and I am not... I [u]again[/u] request that you cite some examples of forged packet used for network management on private networks.[/b]
            Letophoro
          • RE: NAT does not forge packets.

            Letophoro,

            Based on your dialog with George, I don't think he's fully thought this through. The very act of examining my traffic to determine which application I'm using is a privacy violation from my perspective, regardless of the small print. Next thing you know they'll be reading my email. And then to essentially kill a data transfer without even telling me WHY it died is es equally unacceptable. Anyway, thanks for your insight on NAT translations, etc.

            gary
            gdstark13
          • Re: gdstark13

            George may have thought it through. However, he has a tendency to approach things from a network administrator's viewpoint. In this case he seems to believe that protecting the network from users is better than improving the network to serve the users better.

            I'm happy that you found my insights of value. Thank you for the acknowledgment.
            Letophoro
          • pretty much

            What they've done is decided that someone setting up a date using Voip is more important than some music professor trying to collect a long forgotten copy of a symphony through bit torrent.

            Would be similar to a highway department deciding that passenger cars should be banned so that trucks would be better able to transport goods. (after all, most vehicles in tie-ups are CARS right?)

            Traffic control is fine, but control it by limiting my speed, not by blocking traffic.

            And no, I don't normally use any type of torrent software, tried once when trying to DL a linux distro, but for some reason it would NOT connect to anything so I used FTP and uninstalled the torrent software. Makes me wonder if being on comcast has any bearing, and if so how much bandwidth did they save when I downloaded it using FTP instead of a torrent?.

            Ken.
            merc2dogs`
      • re; Every Comcast customer I've talked to contracts

        [i]Every Comcast customer I've talked to contracts your experience. They've had no problems downloading, uploading, or seeding if it isn't excessive. If you limited your upload bandwidth usage and number of connections, it shouldn't be a problem. If enough Comcast customers became seeds at full throttle, it would make the network suffer.
        [/i]

        Then you aren't talking to enough people, because as I said in an earlier post, if you're the sole seeder (not the only scenario that this happens, but it's virtually guaranteed to illustrate the problem) and comcast is in the mood to play monkey er man in the middle, you'll spend more time uploading 0kbs than not (and in this example, there's only one torrent and 1 or 2 downloaders.

        They're not basing this on how the amount of upstream traffic any one person is generating nor are they lowering the the amount I can upload/second.

        This system actually affects those uploading on less popular torrents than those on very popular torrents, because they can't reset all the leechers fast enough to keep the upload at 0 (or perhaps even below the upstream cap on the subscriber).
        notsofast
    • The legality of the torrent traffic is questionable

      Lets be honest, the majority of torrent traffic is illegal content.

      Why should comcast pay to upgrade its pipes to support these pirates?

      And lets not kid ourselves that torrents are for Linux. Yes, Linux distro's are available over torrent. But don't even begin to tell me that even 10% of torrent traffic is Linux.

      More to the point, why, as a Consumer should I pay more for broadband to pay for Comcast to improve its infrastructure to enable the pirates to get their warez faster. No way!

      It is the selfish minority that are causing these problems. And you know what, it wouldn't be as bad if they could just limit their illegal activities outside of peak hours. Surely they could wait an extra day for the free stuff instead of clogging up everyone browsing experience during peak hours.
      Bozzer
      • Not relevant.

        What is relevant is Comcast's actions.

        "More to the point, why, as a Consumer should I pay more for broadband to pay for Comcast to improve its infrastructure to enable the pirates to get their warez faster. No way!"

        If everyone clogged the network with Bible passages, you'd simply complain 'More to the point, why, as a Consumer should I pay more for broadband to pay for Comcast to improve its infrastructure to enable the Biblethumpers to get their Leviticus faster. No way!'

        Seriously though, Comcast's architecture is flawed, and they are trying to stave off the day they actually have to pay to fix things. Unfortunately, their chosen method is of dubious legality.
        Letophoro
      • Ha. Boy. You have one dimensional thinking.

        I agree that Comcast should never have to be forced to provide any specific kind of service...of any particular kind, unless that is, they are a bit of a monopoly. I have no idea about Comcast as its irrelevant where I live, but I know that where I live each kind of available broadband is run by what is close, if not an actual monopoly as its not like you have a choice between 4 or 5 competitively priced companies for DSL and another 4 or 5 for cable.

        As with any situation of this kind, where a company has the luxury of being a "virtual" monopoly due to their inherent control over the infrastructure (as is often the case)they should be pressured, even heavily, to run a system that performs to reasonably exacting standards. With the internet, that has a very special meaning to many many people.

        The internet, for quite a number of years now, has been one of the few places in the modern world that hasn't been regulated to the point of being mundane. Much of the modern world has become a "take what you get" place with little opportunity for the average person to leverage more out of their surroundings simply by learning more about it. The internet has proved to be different in that essentially everyone pays the same for the same service, but the ones who are educated in its use can get way way more out of it because it hasn't been regulated to death yet.

        Most people realize that and have no interest in service providers or the government or business to turn the free spirit of the internet into one more severely regulated part of life. And likewise, if someone wants to be a virtual monopoly in providing such a service they should be forced to compete for that privilege, not regular speed so they can avoid or put of badly needed upgrading.
        Cayble