Comcast traffic management issue before FCC

Comcast traffic management issue before FCC

Summary: Today is the deadline for the FCC call for comments on the Comcast traffic management case brought about by a formal complaint from the Free Press and Public Knowledge.  As a former network engineer who designed networks and servers and as someone who has written extensively on these matters, I thought I would summarize the issues in a clear and concise manner.

SHARE:

Today is the deadline for the FCC call for comments on the Comcast traffic management case brought about by a formal complaint from the Free Press and Public Knowledge.  As a former network engineer who designed networks and servers and as someone who has written extensively on these matters, I thought I would summarize the issues in a clear and concise manner.

Background Independent groups last year found that Comcast was sending TCP RESET packets to BitTorrent seeders at various times of the day to cut back the number of upload sessions they could have.  A BitTorrent seeder is someone who is not downloading but acting as a dedicated and peer-to-peer file server.  BitTorrent downloads or uploads while downloading were not affected.  Various groups complained that this was possibly illegal protocol discrimination using forged TCP RESET packets while Comcast maintained that this was reasonable network management to assure fair distribution of bandwidth to all their users.

The upstream contention problem A typical Cable broadband network such as Comcast operates under the DOCSIS 1.1 standard which offers 10 mbps of upstream bandwidth and 40 Mbps of downstream bandwidth bandwidth shared amongst the neighborhood.  Since the typical user has a static upstream cap of 384 kbps, it would be possible for 26 BitTorrent seeders and/or BitTorrent uploaders to completely jam the upstream pipe rendering the entire network unbearable.  Since a typical Cable broadband company provisions between 50 and 400 users (typically somewhere in the middle) per cable loop, it is possible for ~10% of the users can jam the entire upstream network which ultimately affects downloads as well since services can't be asked for.  This is further complicated by the fact that DOCSIS networks use a reservation system for upstream traffic on a collision network.  Too many requests for upload slots and the requests collide and no one gets to transmit anything.

Accusations of discrimination Some have complained that this was content discrimination.  But Comcast does not discriminate based on content; Comcast discriminates against excessive upstream usage that chokes up their entire broadband network.  The EFF complains that this was "protocol discrimination" against BitTorrent and other P2P (peer-to-peer) applications, but it is a fact that BitTorrent and P2P are the biggest upstream bandwidth users.  Since BitTorrent seeders who only continuously upload throughout the day can be reasonably classified as dedicated servers, they actually fall under prohibited services under Comcast's TOS (Terms Of Service).

Blocking versus delaying Comcast says they're merely delaying BitTorrent seeders from uploading to their peers while their critics say they are blocking.  It is true that Comcast blocks BitTorrent seeds when the broadband network is very busy, but they do allow BitTorrent seeding at most other times of the day.  Network Engineer and Internet pioneer Richard Bennett explained this best in his comment to the FCC that since BitTorrent and other peer-to-peer applications all have the ability to resume transmission at where they left off, temporary blocking of seeders effectively acts as a delaying mechanism.  The file eventually gets served to the remote party outside of Comcast's network intact.

Consumer versus commercial Internet connection The reality is that Comcast customers were never blocked, throttled, or delayed from receiving any services; they were delayed from offering hosting services (BitTorrent seeding) that were technically prohibited to begin with under the terms of service.  Comcast's consumer broadband service technically doesn't have to act as a commercial hosting service to other customers in and outside of Comcast's network so the fact that they permit seeding most of the day seems like a reasonable compromise.  Furthermore, BitTorrent users who are downloading are continuously uploading during the download without any delaying action so it isn't as if Comcast refuses to participate in P2P uploads.

Blocking of Lotus Notes Comcast's network management mechanisms did have a bug in them that accidentally blocked Lotus Notes traffic, but this issue was fixed months ago when the issue was first brought to the attention of Comcast.  All software and hardware implementations have bugs and we expect the service provider to act in good faith and repair the problems as soon as possible.  In this particular case, Comcast appears to have acted quickly and properly by fixing the problems that blocked Lotus Notes.

The complaint to the FCC The Free Press and Public Knowledge filed a formal complaint to the FCC to immediately enjoin Comcast from these network management practices before the merits are decided and the facts weighed.  This is an unreasonable request since Comcast customers would be harmed by network traffic jams due to the lack of any traffic management.  The Free Press and Public Knowledge also demanded fines of $195,000 per infraction which would amount to over $2 trillion dollars if we counted every Comcast customer.  This is obviously impossible since it exceeds the gross revenue of any corporation in the USA.

<Next page - Impractical alternatives proposed>

Impractical alternatives proposed

The Free Press and Public Knowledge acknowledged the need for Comcast to manage their network and they offered some alternatives.  Here is a quote from their petition to the FCC:

More importantly, if Comcast is concerned that the collective set of users running P2P applications are affecting quality of service for other users on a cable loop, they could readily set dynamic quotas for each user on the loop, so as to ensure that there is always bandwidth available for users who are not running P2P applications – and they could do so without interfering in protocol choice. Or they could also charge by usage, provide more bandwidth to all users, or actually offer high symmetric broadband speeds.

The first proposal from the Free Press and Public Knowledge petition is the use of "dynamic quotas for each user on the loop".  This would certainly be an ideal solution, the only problem is that such a mechanism does not exist.  Since we are dealing with upload congestion, the place to solve the problem would be the cable modems at the customer premise.  The problem can't be dealt with at the head unit CMTS (Cable Modem Termination System) since the damage would already be done to the shared Cable medium before the flood of upstream data ever reached the CMTS.  The currently deployed cable modems only have static upstream quotas that can be configured upon boot up, but they cannot be changed dynamically on the fly.

The second proposal from the Free Press and Public Knowledge petition is to "charge for usage" or in other words, a metered Internet service.  The problem is that the Free Press is hypocritical on this proposal since they are actively criticizing Time Warner's metered Internet service.  In this Free Press press release, they state:

"Compared to that approach, Time Warner’s proposal is better — at least customers will know what they’re getting into. But metered prices may chill innovation in cutting-edge applications because consumers will have a disincentive to use them. Viewed in the context of our long-term national goals for a world-class broadband infrastructure, telling consumers they must choose between blocking and metered pricing is a worrying development."

While I'm in full agreement that metered Internet service is a bad idea, I am shocked that Free Press, Public Knowledge, and even the EFF would propose metered Internet service which is something that is so anti-consumer.  It's even more shocking how inconsistent and hypocritical the Free Press is.  Note that the EFF did not join the Free Press and Public Knowledge in their FCC petition, but they echoed many of the same things in their paper "Packet Forgery By ISPs: A Report on the Comcast Affair".  So the very services such as video distribution over BitTorrent that the Free Press, Public Knowledge, and the EFF are asking to be protected are the very services that would be completely shut down under a metered Internet service model.  It would simply be cost prohibitive to run any sort of P2P applications.

The final proposal from the Free Press and Public Knowledge petition is that Comcast should "actually offer high symmetric broadband speeds".  Now everyone wants faster Internet service, but would Comcast's inability to deliver immediately on this constitute a crime and deserve a fine?  If anything, a large fine would prevent them from reaching their publicly stated goal of 20% conversion to DOCSIS 3.0 which would offer a 12-fold increase in upload and a 4-fold increase in download speeds (120 Mbps up and 160 Mbps down).  So what's actually driving this conversion to DOCSIS 3.0?  Verizon's FiOS and AT&T's U-Verse service putting competitive pressure on Comcast in a free market, not Free Press and Public Knowledge.

<Next page - Network management solutions that are practical and fair>

Network management solutions that are practical and fair

The EFF and others have suggested that packet dropping would be a better way to solve the congestion problems and it avoids using forged TCP RESET packets but this is neither practical nor fair.  To illustrate this, I've come up with the following illustration with three technical solutions to the problem of upstream congestion with varying efficacy.

The illustration above is a simplified architectural view of a DOCSIS shared-medium network.  Note that the packet switched portion of the network we call the Internet doesn't really start until the data gets to the CMTS.  That means traditional packet switching traffic engineering doesn't apply on the first hop since the reservation system works on a collision network.  Each house in the drawing represents approximately 10 to 20 homes.  The red homes represent the heavy BitTorrent users who seed and act as dedicated BitTorrent servers with tens of upstream connections while the black homes represent the typical household which uses relatively few upstream connections.

Solution 1: Use TCP RESET packets to reduce the excessive upload sessions from the relatively few users acting as BitTorrent seeds.  Even though we can't say for certain that Sandvine is the exact appliance being used to send the TCP resets, it doesn't matter since this is the general method that Comcast uses.  Since BitTorrent is engineered to grab as much bandwidth in either direction as it can with explicit design goals to thwart network management mechanisms, no formal application-level throttling mechanism is available.  That means the only mechanism that can be employed to throttle BitTorrent is via external informal methods and the TCP RESET is one of them.  Since the application in this case doesn't understand nor would it intend to honor requests to slow down and reduce the number up upstream connections, an alternative method below the application layer is employed.

Solution 2: Use targeted packet drops to only drop or block the upstream BitTorrent packets from BitTorrent seeders.  This method can work nearly as effectively as solution 1, but it's extremely expensive.  A real-time ACL (Access Control List) with thousands of new throttle or block requests per second would put a massive load on any router.  This would require very expensive "fork-lift upgrades" per cable loop.  Solution 1 by comparison is very cheap to deploy since the Sandvine-type appliance only need to run on commodity server and it doesn't need the cooperation of other devices to enforce the rules.  Sending out TCP RESETS is fundamentally cheaper by an order of magnitude compared to targeted packet drops.  It would unnecessarily increase the cost of running the network which ultimately has to be passed on to the consumer with no additional benefits whereas the money would be far better spent on the DOCSIS 3.0 upgrades.

Solution 3: Use random packet drops to reduce overall network congestion.  While this method is cheap to implement and it would ultimately work by slowing every session down equally, it would be extremely unfair to the vast majority of Cable broadband subscribers.  As British Telecom Chief Researcher Bob Briscoe put it in his paper to the IETF (Internet Engineering Task Force),

"It’s equivalent to claiming food rations are fair because the boxes are all the same size, irrespective of how many boxes each person gets or how often they get them."

Proponents of random packet dropping are essentially saying that everyone must slowdown on upload traffic regardless of how many sessions or how little bandwidth they use by some fixed percentage.  To enforce this, everyone's packets would have a certain percentage dropped which would force their TCP/IP stack to slow down.  This would have disastrous consequences on protocols like VoIP since dropped packets mean dropped words within a conversation.  It matters not that VoIP users aren't constantly active or that they only use one session at a time and only requires a small amount of bandwidth while they're on a call, they'll be punished just the same along with the excessive bandwidth and session users.  This is a reckless and unfair approach that should be disregarded.

Conclusion Given the fact that the petitioners don't actually dispute the right of Comcast or any other network provider to reasonably manage their network and throttle down excessive uploaders, the only thing in dispute is the methodology used to achieve these goals.  I've highlighted all the proposed alternatives along with the pros and cons of each solution and I'm confident that the one already in use by Comcast is the least intrusive, least expensive, and most practical way of dealing with the realities of a DOCSIS 1.1 broadband network.

Market forces in the form of competition are forcing Comcast to upgrade to a much higher performance and more symmetrical DOCSIS 3.0 network and that is a win for the consumer.  But regardless of how fast the network is, there will always be a need for practical network management solutions that ensure per-user fairness and not per-session fairness.  The last thing we should do is force Comcast to implement more expensive and/or less fair traffic management schemes that at best wastes money and at worst degrade performance for consumers who are using far less than their fair share of bandwidth.

<Return to top>

Topics: Verizon, Broadband, Government, Government US, Hardware, Mobility, Networking, Telcos

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

104 comments
Log in or register to join the discussion
  • As i suspected...

    This is just what i suspected, Comcast is using antiquated equipment (DOCSIS 1.1) when other ISPs have been using DOCSIS 2.0 for YEARS....and charged much better rates than Comcast. With the 54.99+ per month that Comcast charges each sub, they should be funneling that in to upgrade their network, NOT rely on TCP reset packets to do their job for them. TO THE COMCAST CORPORATE FOLK THAT READ THIS: Take a look at your competition and what they are doing, you should follow suit. Over charging your customers and delivering them a POOR internet experience is not the way to ensure you keep your marketshare once you experience a little thing called COMPETITION.
    JT82
    • Two errors, DOCSIS 1.1 is common, and they don't charge $55 per month

      Two errors.

      DOCSIS 1.1 is common in the Cable broadband world. Comcast has promised to have 20% of their network transitioned to DOCSIS 3.0. They will funnel their money in to upgrading their network so long as we don't have these frivolous 2 trillion dollar fines being demanded to the FCC against Comcast.

      Comcast does not charge $55 per month.
      georgeou
      • Pricing varies by market...

        but in the Salisbury, MD and surrounding markets, for HSI only yes is 54.99. I still think there should be some punitive damages against Comcast for what they are doing because they still have grossly oversold the loop and therefore created the problem for themselves. Cox communications (in SE VA) can manage their network and no way is it still DOCSIS 1.1 because they have easily double the traffic/customers and do not resort to these low tatics.
        JT82
        • All ISPs over provision, some more than others

          All ISPs over provision, some more than others. There is no precedent for fining companies for over-provisioning.

          I?m perfectly willing to agree with you that more transparency is needed and more competition, but a lot of the problem is the regulation and build-out requirements. All those regulations actually make it HARDER for smaller startups to get in to the game because they?ve been priced out by competition.
          georgeou
          • Just as airlines over book...

            It makes sense to over provision to a degree just as airlines overbook flights because there are always the folks who just "dont show up" or in this case such a minimal user, their impact is negligable. But when you consistantly have customers getting speeds of 756kpbs MAX in when you are provisioned for 5Mbps during "peak" hours..there is a problem (and believe me they do not want to hear it, ive tried) and they simply grossly over provisioned the loop. I dumped them for DSL and never looked back.

            Yes I agree the numerous regulations and requirements do hurt the smaller startups and maybe something should be done from that point so that we can start putting the squeeze on the monopolies like Comcast, Cox, and even in my current market Verizon DSL -- and the prices show it..they want to charge over 40.00 a month for 3Mbps dsl....

            Bottom line is in MY opinion, the TCP reset idea is bad. There are much more practical ways of managing your network (maybe dont take on as many providers or where the loop space is tight, upgrade those to the new DOCSIS version). There is a lot of legitimate products that use P2P technologies (World of Warcraft downloads, Skype, etc) and unless they can prove to me that I am not targeted by this TCP reset using these programs? They are simply discriminating against a specific protocol and should be fined accordingly.
            JT82
          • Again, there are no better solutions other than a network upgrade to DOCSIS

            "Bottom line is in MY opinion, the TCP reset idea is bad. There are much more practical ways of managing your network (maybe dont take on as many providers or where the loop space is tight, upgrade those to the new DOCSIS version)."

            Again, there are no better solutions other than a network upgrade to DOCSIS 3.0. As I pointed out in this in-depth blog, all the proposed solutions are either pie-in-sky or they actually hurt you more than they help you. How would you like to be playing a game and have your packets dropped even though you're low-bandwidth and single session because some guy at EFF thinks it's a great scheme to punish everyone across the board? How would you like to get the EFF recommended Australian plan with metered Internet service?

            Becareful of what you wish for because you may just get it. The only true solution is an upgrade to DOCSIS 3.0 and everything else just distracts from that goal.
            georgeou
          • Then what are they waiting on..

            Comcast is supposed to be the LARGEST cable internet provider (i think i heard it in their propaganda somewhere) so if they are only committing 20% of their network do DOCSIS 3.0...wheres the other 80%? For the prices they charge, this rollout shouldnt be THAT big of a deal. Granted, I personally do not know if its simple software or some hybrid software/hardware approach, i never worked on cable internet systems -- but still they should have done this YEARS ago (or done a step to DOCSIS 2.0 as a bandaid solution).

            And the EFF's idea of "random" packet dropping and metered service BOTH suck (to be frank). Metered internet does nothing but HURT the consumer because we have to pay for every bit, regardless if we wanted it or not. The random dropping of packets could BE possible, but still kind of suck because would it appropriately read the tagging and protocol headers and NOT injure services like VoIP (which i personally rely heavily on at home). Everything I do communication wise is via a HSI pipe and to have that hampered with is irresponsible on the part of the provider. As consumers we pay for a specific service (granted it does say otherwise in the TOS) and they should deliver or take the required steps to demonstrate due diligence to the subs/network, in this case would be to upgrade areas to DOCSIS 2.0 where they had issues similiar to what I was experiencing on the loop.
            JT82
          • I hear what you're saying, but the current fiasco isn't helping

            I hear what you're saying, but the current fiasco before isn't helping with the proposals from the Free Press, Public Knowledge, and EFF. What we need is reasonable debate and sensible solutions.

            Upgrading to DOCSIS 2.0 is out of the question, it's easier just to upgrade to 3. We need more competition to push the pace.
            georgeou
          • Or something else to kick them in the butt...

            With the little knowledge I have of the DOCSIS rollout dates, the 1.1 version that they are primarily still are using was finalized back in April of 1999...tell me why nearly 9 years after this was finalized they havent found any way to upgrade? DOCSIS 2.0 was finalized back in 2001 and the latest one 3.0 released back in august of 2006. Even from a network management point of view they are not exercising due dilligence to the consumer by employing the very latest and greatest to their customer. They should have no right to complain about the bandwith that P2P uses because they lacked the motivation to upgrade their network.

            Yes upgrading to DOCSIS 2.0 now would be foolish and a waste of resources, BUT this should already be done, considering it was released 6 years ago.I am not a big fan of government intervention but if competition cant do it, our Uncle should :)
            JT82
      • re: Two errors

        Good info George, thanks.

        btw Comcast internet cost is $42.99/mo here, with an extra $10.00/mo if you go for the higher speed package. That may be what he was referencing when he said $55.
        Badgered
      • Common?

        Do you have numbers?

        I think most of the other ISPs have already switched to newer technologies - the last two places I've been to offer 5 mbps and 8 mbps (downstream) plans. I haven't seen less than 1 mbps for a single household in a long time.

        Granted, I may be a statistical outlier, but I'm suspicious of your claim that 1.1 is still "common." I think most other companies have moved to at least version 2.
        CobraA1
      • They charge MORE than $55

        This is right from Comcast's page for a Boston zip code:

        High-Speed Internet for Non-Comcast Cable Customers $57.95
        MGP2
  • Abuse of Cable Networks

    I think the main problem is the average joe is not violating the cable network or his loop, but he is the one that has to suffer when he just wants to go look at CNN or whatever.

    Most likely the uploaders are violating some kind of policy or are doing some kind of illegal activity, or are running a business or using their connection for a commercial endeavour such as working from home.

    Some examples of this might be running a Game Server.
    Opening your 2,000,000 songs for upload to other people.
    Running a Website from your house.
    Mass mailing software.
    Uploading Internet Porn.
    Freelance computer programers and animators working from home.
    Your computer is a Zombie!
    Spyware.
    Your wireless modem has 100 connections.

    There are probably a million other things that could cause a red flag.

    Just keep in mind I remember these commercials promising up to 50 times faster than Dial-up. So this is the case of the provider ISP not managing their netwrok properly. Then they have all kinds of excuses.
    ceh4702
    • And it will continue

      Until a good percentage of the customer base stop becoming sheep to the ISP's continious lines of BS and drop them like a bad habit. Unfortunately, the HSI pipes have integrated themselves into our lives so greatly they are becoming a utility rather than a luxury item..and they know this. Competition needs to come in and balance out this market. What ever happend to Broadband over Power Lines? Even if it is only 3.0mpbs max, who cares..thats enough for the average user to support a few wireless clients and multiple VoIP calls.
      JT82
      • On Broadband over Power Lines....

        ...I graduated from a EE school that specializes in Power Distribution, so I've kept in touch with some of the professors and here is what they tell me about Net over Power Lines.

        1) It's a lot more costly to implement than the utilities originally were told/thought. It turns out that to RELIABLY provide high speed to every socket on the planet, special bypass filters have to be installed on every transformer coil terminal on the transmission path. The filters are cheap, the labor to install them is NOT.
        2) Power companies already deal with a rather difficult group of employees that they call "engineers". The management guys feel that bringing in another group of them to run the network side will be overly expensive and just not worth the trouble.
        3) Certain "groups" have managed to "convince" the power utilities that it might not be financially sound to move into that market. *wink wink, nudge nudge*
        Take that statement with whatever conspiracy theory slant you want.....
        4) Power Utilities are regulated monopolies and as such can only make SO much money before the regulatory agencies start telling them they have to lower the price they charge customers. Rather than wanting to make a boat load more money, they would rather slightly increase profits and spend it on stuff they can claim are "legitimate" expenses. For example, the CEO of Entergy (our Power Utility here), actually lives near Jackson, MS and has a company helicopter that flies her back and forth to New Orleans (200 miles one way) every day....that is a legitimate business expense for them.
        DCMann
    • Abuse of network?

      So, uploading internet porn is abuse of a cable network? Personally, I upload sex stories that I have written to various websites, and that takes very little bandwidth.
      I also download porn using BitTorrent, and my neighbors who are on the same pipe have told me that they have NO problem getting to their websites while I am doing my thing.
      On the 'running a website from your home'.... Comcast has a service where you are allowed to host a website, with up to 1gb of content, on their computer.... so no one should have to run a website from their home.

      Mass mailing software..... No one I know uses them at all.
      Zombie problems..... Sorry, but Comcast should contact me if they notice that my computer is sending out information in a known zombie-like way.
      Spyware.... those usually only connect once a day (to make it easier to hide them), so that wouldn't take up much bandwidth and usually just sends a list of typed items on your computer and a list of your websites visited.
      Running a game server..... Actually, Comcast's ToS specifically allows that, or at least it did, so that isn't a violation of the ToS.

      Putting 2,000,000 songs for upload to other people.... excuse me, but I know some people who do that, and they have, at most, 5,000 songs, and some of them have stopped doing that now that services like Yahoo Music Unlimited and Rhapsody have come out.

      We really need to start realizing that the problem is not 'bandwidth hogs', the problem is that the companies in question are not putting money into building more pipes and other things, so the internet is getting congested.
      Leria
    • Why do you still try to claim that downloads were affected?

      "but he is the one that has to suffer when he just wants to go look at CNN or whatever."

      You make the same mistake here as many people here, only BitTorrent seeders (dedicated uploaders) were affected during short periods of time throughout the day. Downloads of any kind were never affected. Can we please stick to the facts here and accept even the EFF, Free Press, and Public Knowledge's admission that at least downloads were never affected?
      georgeou
      • P2P downloads need to get from off-Comcast?

        Wait a minute... Although downloads from (say) CNN wouldn't be affected, wouldn't P2P downloads still be affected? You wouldn't be able to get the file from a person on your own (Comcast's) network, so wouldn't it be more latent to grab a file from someone further away?
        AySz88
        • CNN doesn't use BitTorrent

          CNN doesn't use BitTorrent, that's the first thing.
          CNN or anyone else doesn't have a "right" to use the scare upstream capacity of a consumer DOCSIS network to distribute their content. That scare 10 mbps upstream capacity divided among 450 users is for commercial broadband usage, it wasn't designed a commercial file distribution network.

          Forcing Comcast users from downloading from further upstream (could be a head-end cache on Comcast's network or somewhere else on the network) helps alleviate the last-mile scarcity problem to prevent a network meltdown on the upstream in the local cable loop.
          georgeou
          • Typo correction

            "Forcing Comcast users from downloading from further upstream"

            should be

            "Forcing Comcast users to download from further upstream"
            georgeou