Memcached DDoS: The biggest, baddest denial of service attacker yet
Security
We've been seeing a rise of ever bigger Distributed Denial of Service (DDoS) attacks for years now. But, now a new attack method, Memcrashed, can blast your site with over a terabyte of traffic. Good luck standing up to that volume of abuse!
Memcrashed works by exploiting the memcached program. Memcached is an open-source, high-performance, distributed, object-caching system. It's commonly used by social networks such as Facebook and its creator LiveJournal as an in-memory key-value store for small chunks of arbitrary data. It's the program that enables them to handle their massive data I/O. It's also used by many to cache their web-server-session data to speed up their sites -- and that's where the trouble starts.
All too often, incompetent system administrators (sysadmins) have exposed memcached-enabled servers to the internet. Memcached was never, ever meant to be available over the public internet. It has no authentication, so it's easy to abuse.
What happens when an attacker finds memcached on the internet is they can use it to power a DDoS UDP-based reflection attack vector.
And, what does that mean? Let me tell you.
When a server receives a memcached get request, it collects the requested values from memory to form a response. It then sends over the internet in an uninterrupted stream of multiple UDP packets, each with a length of up to 1,400 bytes. That's bad, but there's more.
The attacker can insert records into the open memcached server. By default, memcached uses a limit of 1MB per stored value; however, an attacker can insert even large values since any user can configure it. Remember what I said about "no authentication"? Adding insult to injury, the attacker can demand multiple or duplicate keys with a single request.
See where this is going? This attacker can load large values into the data store and then use them in attacks. So, even with just a single megabyte stored value, the attacker uses a spoofed UDP packet request to ask for that 1MB of data to be sent hundreds of times per request over memcached's default 11211 UDP port. According to the content delivery network (CDN) Akamai, "This would result in a massive amplification factor where a 203 byte request results in 100MB response of reflected traffic."
How bad can it get? CloudFlare, the web-performance and security company, found "15 bytes of request triggered 134 KB of response. This is amplification factor of 10,000x! In practice we've seen a 15 byte request result in a 750kB response (that's a 51,200x amplification)." That's bad. The only practical limit is how much traffic can a single vulnerable memcached server's network tolerate.
Of course, an attacker won't stop with just one vulnerable server. No, they'll push multiple memcached servers to attack their target. The result is an attack that can take down pretty much any server on the planet. With over 91,000 open memcached servers, according to a quick Shodan search, the potential for massive DDoS attacks has never been higher.
So what can you do?
Well, the real answer is for everyone running memcached outside their firewall to stop it. Right now.
While you're at it, if you're running memcached, according to SANS you should harden your memcached server by taking the following steps:
- Open /etc/memcached.conf in a text editor
- Locate the -m parameter
- Change its value to at least 1GB
- Locate the -l parameter
- Change its value to 127.0.0.1 or localhost
- Save your changes to memcached.conf and exit the text editor
- Restart memcached
If you're running memcached, you should also disable UDP support if you are not using it. If you're a developer, as Cloudflare succinctly puts it, "We've been down this road so many times. DNS, NTP, Chargen, SSDP, and now memcached. If you use UDP, you must always respond with strictly a smaller packet size then the request. Otherwise your protocol will be abused."
As a potential victim, you should also close off port 11211. You should also look into DDoS mitigation services from such companies as Akamai, CloudFlare, and Incapsula. Oh, and pray too. When DDoS attacks reach this level of sheer volume, you're going to feel the pressure no matter what you do.
In the long run, as the sloppy maintained memcached servers are brought to heel, this DDoS problem will go away. Until then, we're going to suffer massive DDoS attacks. Just ask Github.
Related Stories: