BLOG

Making Sense of the Last Month of DDoS Attacks

David Holmes 缩略图
David Holmes
Published October 27, 2016

In this piece, we’ll take a brief look at the impact of the unprecedented #DDoS attacks of September and October 2016 and provide guidance for those interested in improving their resiliency.

Let’s celebrate that the Internet of Things has arrived! We know it’s finally here because it got used as a cyber weapon three times starting in September. It started with a massive attack on blogger journalist Brian Krebs, whose website krebsonsecurity.com was hosted and protected by the CDN Akamai. The attack was severe enough to affect Akamai’s other customers, so they dropped their pro-bono defense of the blogger.

Industry researchers (including F5’s) were alerted to a new botnet comprised of DVRs and video cameras and other “things.” But the DVRs and video cams are significant because they have both high CPU capability (which can therefore host sophisticated malware) and high-bandwidth uplinks (from which they launched attacks up to 100 Mbs each). The total attack traffic directed at Krebs was 620 Gbs, which at the time was the world’s largest DDoS attack.

The second attack knocked out OVH, the largest French hosting provider (number three in the world, if you believe the literature) for much of a day. Many media reports listed the OVH attack at nearly double the size of the Krebs attack; a full terabit per second.

The third and possibly worst attack came against Dyn, the DNS services company. Dyn is the DNS provider for many marquee websites, including Twitter, Spotify, and GitHub (where ironically the IoT botnet code was posted).

So, Who Did It?

By strange coincidence, industry luminary Bruce Schneier had been making the interview circuit the previous month warning of massive, nation-state DDoS attacks to come. His inside information came from unnamed industry sources who were spotting calibration DDoS attacks against the core Internet infrastructure.

But, as it turns out, those probing attacks may have been just coincidence. Schneier himself admitted on his blog that he doesn’t think the Dyn attack was a nation-state at all. But maybe that nation-state is just watching and waiting.

The key to attribution in the Krebs, OVH, and Dyn cases may likely come from Brian Krebs himself.

“The attack on DYN comes just hours after DYN researcher Doug Madory presented a talk on DDoS attacks in Dallas, Texas at a meeting of the North American Network Operators Group (NANOG). …the record 620 Gbps DDoS against KrebsOnSecurity.com came just hours after I published the story on which Madory and I collaborated.”

Very likely the adversary is an individual (or small group) with an agenda against Brian Krebs or Doug Madory, or both.

We could all understand that millions of users would be affected by a full-scale China-US cyberwar. But if it turns out to be a personal agenda from a single individual against another single individual? That’s even scarier than a nation-state attack. What kind of Internet have we built when a single person with a grudge against another person can interrupt critical services for millions of people?

How Did We Get Here?

F5 has been monitoring the hunt for Internet of Things (IoT) devices for over a year now. Our first report on this issue was published in July of 2016 and showed a 140% increase in year-over-year telnet and SSH brute force scans. Telnet and SSH are widely used remote administration ports, and they are often left unknowingly exposed to the Internet with vendor default (or easy to guess) user name and passwords.

The unprecedented DDoS attacks on Krebs, OVH, and Dyn in October of 2016 used exactly this technique (scanning for telnet ports and vendor default passwords on IoT devices) to create a botnet of unique capability.

Everyone’s a target again. The bot herders aren’t afraid to turn their cyber weapon against some of the largest providers in the world—targets that were previously thought untouchable.

While these attacks might feel like they happened overnight, in reality, the bot herder has been slowly searching for, finding, and compromising vulnerable IoT devices for at least a year.

Deep Dive into What We Know

We actually have a pretty good idea of the capabilities of the IoT botnet, Mirai. The researchers at F5 wrote a technical breakdown of the Mirai bot shortly after its author released the source code on hackforums.net.

The collective firepower is likely an order of magnitude greater than previous botnets; north of terabits per second.

The IoT botnet includes (but is not limited to) the following advanced DDoS techniques. The prescriptive guidance section below will provide recommendations on how to mitigate these, when possible.

HTTP GET Floods Resistant to Redirection

HTTP GET floods were already pernicious. For years, attackers have been able to disable web sites by sending a flood of HTTP requests for large objects or slow database queries. Typically, these requests flow right through a standard firewall because they look just like normal HTTP requests to most devices with hardware packet processing. The Mirai attack code takes it a step further by fingerprinting cloud-based DDoS scrubbers and then working around any 302 redirects that the scrubbers send back. Redirects used to be a good way to stymie simple bots, but this one isn’t simple.

DNS Water Torture

The Mirai bot includes a “water torture” attack against a target DNS provider. This technique is different from the regular DNS reflection and amplification attack because it requires significantly fewer queries to be sent by the bot, letting the ISP’s recursive DNS server perform the attack on the target authoritative DNS server. In this attack, the bot sends a well-formed DNS query containing the target domain name to resolve, while appending a randomly generated prefix to the name. The attack becomes effective when the target DNS server becomes overloaded and fails to respond. The ISP’s DNS servers then automatically retransmits the query to try another authoritative DNS server of the target organization, thus attacking those servers on behalf of the bot.

Dyn confirmed in their blog post, Dyn Analysis Summary of Friday, October 21 attack, that the water torture was indeed used against them.

“For example, the impact of the attack generated a storm of legitimate retry activity as recursive servers attempted to refresh their caches, creating 10-20X normal traffic volume across a large number of IP addresses. When DNS traffic congestion occurs, legitimate retries can further contribute to traffic volume.

It appears the malicious attacks were sourced from at least one botnet, with the retry storm providing a false indicator of a significantly larger set of endpoints than we now know it to be. We are still working on analyzing the data but the estimate at the time of this report is up to 100,000 malicious endpoints.”

F5 researcher Liron Segal had detailed the mechanics of the bot’s Water Torture attack in his post weeks earlier—Mirai, the Bot that took down Krebs.

Updated Layer 4 Attacks

According to the bot’s creator, the so called “TCP STOMP” attack is a variation of the simple ACK flood intended to bypass mitigation devices. While analyzing the actual implementation of this attack, it seems that the bot opens a full TCP connection and then continues flooding with ACK packets that have legitimate sequence numbers in order to hold the connection alive.

Given our understanding of the individual threat vectors with the new IoT bot, we can provide some guidance for the mitigation of those individual threat vectors.

Let’s be clear before we start the guidance. The Krebs, OVH, and Dyn attacks are in a class by themselves. Clearly, existing DDoS mitigation techniques did not easily throw back the attackers—hence the outages and the headline media coverage. However, mitigation eventually did take effect in many cases. And proper architecture, such as using Anycast and dispersed data centers helped, as well. Note that the West Coast and western regions of the US were largely unaffected by the Dyn attack.

Prescriptive Guidance

Our guidance comes from our own customers. F5 has been delivering applications for the world’s marquee brands for twenty years, and many of those customers get attacked with DDoS every day. Our most experienced customers break their defenses into three or four zones:

  • Cloud scrubbing
  • Network defense
  • Application defense
  • DNS, where applicable

A superlative DDoS-resistant architecture therefore looks like this:

This is the DDoS protection reference architecture, which has been widely used by F5 customers for years. The full reference architecture and recommended practices can be found at F5.com. However, for faster consumption, the guidance relevant to these attacks is detailed below.

Volumetric Defense

The French hosting company OVH was hit with a volumetric attack of 990Gbs. There were reports that the Dyn attack peaked at 1.2 terabits. Attacks of that size can usually only be mitigated by cloud scrubbers that specialize in defense at scale. Cloud scrubbers, including F5's Silverline DDoS Protection, intercept the attack traffic, scrub it clean, and send only the good traffic to the target over pre-arranged tunnels.

Guidance: Organizations should ensure they have agreements with one or more cloud scrubbers prior to getting attacked. Configuring the pre-arranged tunnels is not something that can easily be done in the midst of a volumetric attack. Contract with a cloud scrubbing DDoS defense as part of your DDoS strategy.

Guidance: Attack diffusion can be your friend for volumetric attacks. Remember that DNS can be your friend, too; Anycast your global data centers for replicated content to diffuse DDoS attacks when they happen. Each data center that participates with Anycast can help divide the attack.

Relevant Materials:

Network Defense

The Mirai bot includes several layer 4 attacks in its arsenal: standard SYN floods, TCP floods, and UDP floods. These ancient threat vectors must be mitigated either at a cloud scrubber or, if they are sufficiently small, at the network defense tier in the data center. The network defense tier is built around the network firewall. It is designed to mitigate computational attacks such as SYN floods and ICMP fragmentation floods. This tier also mitigates volumetric attacks up to the congestion of the ingress point (typically 80 to 90 percent of the rated pipe size).

Guidance: Many firewalls are not resistant to DDoS attacks unless properly configured. Check with your network firewall vendor for settings. Some customers will put anti-DDoS devices in front of their firewalls to repel layer 4 attacks.

Guidance: The F5 firewall module (BIG-IP Advanced Firewall Manager (AFM)), was designed specifically to repel layer 4 attacks. Some architects use BIG-IP AFM for just this case—either in front of, or replacing conventional network firewalls. Hardware appliances with AFM use field programmable gate arrays to repel more than 30 types of packet floods and offload the work from the CPU.

Relevant Materials:

Application Defense

The Mirai bot can generate impressive HTTP GET floods and handle directs. Because GET floods look like normal traffic to the network defense devices, they must be handled at the application tier. GET floods are by far the most common application layer attack type that F5 sees, and there are many ways to mitigate them, depending on the product portfolio that a customer is wielding.

Guidance: For bots that can handle simple redirects, F5 recommends either throttling connections based on their request-per-second metric or by using what is called a “login-wall.” A login wall requires a connection to be authenticated to the application before it can consume non-cached or dynamic resources such as database queries.

Relevant Materials:

DNS Defense

There are two DNS issues worth talking about in regards to the Dyn attack. The first and most straightforward is what to do if your DNS provider is taken offline by one of the new types of attacks. Dyn was the name-service provider for Twitter, GitHub, and Spotify, so when Dyn was blocked, end-users were unable to find IP addresses for these services.

Guidance: Build resiliency into your DNS plan that includes, but is not limited to, multiple DNS providers to serve addresses for your critical applications. In this way, if one of the providers succumbs to an attack temporarily, the other provider can serve your addresses. It may slow your end users by a few milliseconds, but your applications and services will still be available.

The second issue is what to do if your own DNS server comes under attack. DNS is the most targeted service with HTTP being second. When DNS is disrupted, all external data center services (not just a single application) are affected. This single point of total failure, along with the often under-provisioned DNS infrastructure, makes DNS a tempting target for attackers.

Recall that even if your own server isn’t under attack, an outage downstream of you could cause a different set of DNS servers to flood your DNS servers with requests as they try to fill their own caches. Dyn reported 10 to 20 times normal requests when this happened with them, and that was from legitimate DNS servers trying to cope with the situation.

Guidance: A significant percentage of DNS services are under-provisioned to the point where they are unable to withstand even small-to-medium-size DDoS attacks. DNS caches have become popular as they can boost the perceived performance of a DNS service and provide some resilience against standard DNS query attacks. Attackers have switched to what is called “no such domain” (or NXDOMAIN) attacks, which quickly drain the performance benefits provided by the cache.

For F5 customers, F5 recommends front-ending the DNS services with the DNS proxy module called F5 DNS Express™. DNS Express acts as an absolute resolver in front of the existing DNS servers. It loads the zone information from the servers and resolves every single request or returns NXDOMAIN. It is not a cache and cannot be emptied via NXDOMAIN query floods.

Guidance: Remember that DNS can be your friend during a DDoS attack; Anycast your global data centers for replicated content to diffuse DDoS attacks when they happen.

Guidance: Consider the placement of DNS services. Often the DNS service exists as its own set of devices apart from the first security perimeter. This is done to keep DNS independent of the applications it serves.

Some large enterprises with multiple data centers serve DNS outside the main security perimeter using a combination of BIG-IP DNS with DNS Express and the BIG-IP AFM firewall module. The main benefit of this approach is that the DNS services remain available even if the network defense tier goes offline due to DDoS.

Relevant Materials:

Conclusion

The Krebs, OVH, and Dyn attacks mark a new phase in DDoS. At the same time, they mark the coming of age of the Internet of Things.

As you can see, F5 has a lot of experience researching, combating, and writing about DDoS and we want to work with customers to keep their applications available. This will require vigilance on all our parts—from F5 to partners to customers.

If you are a customer, or even if you aren’t, and you come under attack, remember that F5 Silverline DDoS Protection is only a phone call away: 866-329-4253.

As for the IoT threat, blackhats share with each other all the time (they shared Mirai after it attacked Krebs and OVH, and it was used in the Dyn attack). We in the InfoSec community need to take a page from their playbook by banding together to solve this global IoT problem. We have no choice but to do this. It’s human nature to understand problems, fix, and therefore evolve.

There will no doubt be hiccups over the next few years while attacks grow in size, scrubbing services grow in bandwidth to accommodate these large attacks, and IoT device manufactures figure out how to deal with their device insecurities. Organizations and consumers have to get used to this evolving threat, like all other major issues before this one.