Anatomy of 2,000 Compromised Web Servers used in DDoS Attack

This post is available in Spanish (Este post está disponible en español).


One of our clients was being attacked by a layer-7 DDoS attack for more than a week. The attack was generating around 5,000 HTTP requests per second, which took his site and server down. It also caused his hosting company to suspend his server for “ToS violation”. Yes, some hosting companies consider a ToS violation if you are suffering a DDoS. It is mostly an excuse to protect their networks, but very annoying for someone victim of an attack.

After a week of pain, he found our Website Firewall (WAF) product, the rest as they say is history. We were able to quickly block the attack and restore his site to normal operations. If that was all that there was to the story, then many would find this to be a very uninteresting story.

A Diamond in the Ruff

As is customary in our lab, we began analyzing the attack to see if there was anything else we could learn. That is when we noticed something curious, the IP addresses hitting the server were always constant.

We did some operating system identification (using p0f) and the attack was coming mostly from web servers running on Windows and Linux:

Sucuri - Web Server Compromise - Operating System Distribution

This is a bit unusual, most layer-7 DDoS attacks leverage compromised desktops and very few of them actually run on Linux. We also started checking the banners of these IPs and we saw a variety of web servers, but the majority of them were running Apache:

Sucuri - Web Server Compromise - Makeup Distribution

There were also a number of other IP addresses that were not displaying the server banner (or too slow to respond to our HTTP queries). This is the breakdown of the most used web servers on the IP addresses attacking our client:

Sucuri - Web Server Compromise - Web Servers

These were but a subset of the most active, in total we found close to 2,000 different IP addresses causing the damage (each one hitting the server a few times per second).

As far as location, most of them were coming from China, Taiwan and Thailand:

Sucuri - Layer 7 DDOS Attack  Source

Another interesting point is that more than 300 of them were using “AppServ Open Project” (version 2.5.9 or 2.5.10), which unfortunately bundles an old and outdated version of PHPMyAdmin.

Compromised Web Servers

From what we could gather, it seems someone created a bot net of compromised web servers that are running AppServ, outdated Apache, outdated IIS and other vulnerable software (e.g., PHPMyadmin).

This setup gives the attackers good power when attacking sites as they remain anonymous behind proxies. In this scenarios, they specifically focused on Layer 7 HTTP flood attacks, but it could have as easily been DNS application, an SSDP attack or any number of the available options when it comes to DDoS.

We will be contacting the network/hosts responsible for them to see if we can get them patched or shut down.


If your are a victim of a DDoS attack and need help, let us know, we’d love to help, you can start here.

Quick Analysis of a DDoS Attack Using SSDP

This post is available in Spanish (Este post está disponible en español).


Last week, one of our many clients came under an interesting attack. Enough that it was flagged for human intervention. The interesting aspect of the case was that it was a multi-faceted DDoS attack.

The first issue we noticed was a Layer 7 – HTTP Flood (DDoS) Attack attack generating thousands of HTTP requests per second. This is no uncommon, and we’ve written about several instances of this in the past. When they are large enough they trigger a number of system alarms that allow us to quickly run counter intelligence based on the logs and to create custom patterns, signatures, that we can then deploy to and proactively protect our clients.

Once the Layer 7 DDoS attack was under control, we continued our investigation of the server and noticed that it was also suffering other types of DDoS attacks. Our attention turned to tcpdump. If you’re not familiar with TCPDUMP, it’s a command line packet analyzer that allows you to intercept and display all traffic that is hitting your computer. This allowed us to further investigate what the attacker was doing; further inspection revealed a large number of UDP packets hitting the server. Since we don’t run UDP on that server, it was easy to deduce that it was a DDoS attack.

If your site is currently experiencing these problems, get in contact with us. We can help and it’s helpful to see different iterations of these attacks in the wild. If you’d like to read more about DDoS attacks, you can do so here or here.


Read More

CloudProxy + SPDY = A Faster Website

Our CloudProxy Firewall already protects and speeds load times for 1,000’s of websites. Now, it’ll be even faster. We’re happy to announce that we just added support for SPDY (pronounced speedy) across all of our plans and servers. Any website being protected by our CloudProxy firewall can enable SPDY support with just one click:

SPDY

If you haven’t heard of SPDY (SPeeDY), it is a new protocol developed primarily by Google for transporting web traffic. It reduces page load latency through compression, multiplexing, and prioritization. In non-technical terms, it makes your HTTPS site a lot faster and it is supported by all major browsers, including Chrome, Firefox and Opera.

Internet users using these browsers can take advantage of this protocol on sites that have SPDY enabled (like Google.com, Twitter.com, etc…).

We’re excited about this because while we continue to protect our clients’ websites with our WAF, we will also be helping to make their sites faster and more reliable.

If your site is already being protected by CloudProxy, just login and enable SPDY. If you haven’t yet protected your website, head to our CloudProxy homepage to see why 1,000’s of clients are using our firewall to shield their site from attacks.

Website Firewall Update – Introducing 2FA and More

Today, we are launching the new and improved Protected Page capability in our Website Firewall, CloudProxy. It allows for a simple (1-click) activation of secondary authentication methods on any page of your site. It means you can easily add the following to any page on your website:

  • A custom password verification
  • Two Factor authentication (2FA) using Google Authenticator
  • Captcha verification
Protected Page

Have you ever needed to protect a specific page on your site with a custom password or using two-factor authentication? In the past, it required a lot of coding and messing with plugins that aren’t always easy to setup.

We’re happy to say that we’ve made it a lot easier with the Protected Page feature. In your CloudProxy dashboard, you can specify the location you want to protect (/wp-admin for WordPress or /administrator for Joomla, for example) and choose how you want to add a second layer of protection to it:

cloud-2fa

We are really excited about these options and hope you are too!

Use Cases

You can use it on any website, with any CMS or web application and it doesn’t require and coding or other change to your website. Here are some of the ways that clients will be able to take advantage of this upgrade:

  1. Want to prevent bots from submitting comments? Or filling a form on your site? Add the captcha option.
  2. Want to restrict access to /wp-admin, /administrator or any other admin page on your site? Add a custom password or two-factor authentication.
  3. Want to make a page, private? Add a password to it.
  4. Do you have an employee portal, web mail or custom application that you want to restrict access? Add a password to it.

There are many ways you can use this new functionality on your website. Interested? Log into your CloudProxy dashboard, then go to security and then to the Protected Page section to start using it right away.

Sucuri CloudProxy – Website Firewall Enhancements

When LA’s DA says that, “73% of our local businesses appear to have been hacked,” it begins to illustrate the importance website protection will play in the future of business, which is why we’ve placed so much emphasis on website protection on this blog over the last few months.

Protection is no longer a, “nice to have,” and has crossed into the realm of necessity. Website owners know about website hacks and DDoS attacks and malware injections, but they often don’t know how to stop them from happening and until a hack hurts their own business, it’s very easy to believe that these hacks will happen to other people and other businesses. That’s why we’ve written so much about our Website Firewall – CloudProxy lately. Truly, we want to help keep your website safe.

In that spirit, we challenged ourselves to make our firewall more intuitive to use so that any website owner will be able to take control of their own security protocols. We’re proud to announce that our team has made some great strides, in terms of user experience, lately and, in this post, we’ll highlight a few of the enhancements we’ve put in place.

CloudProxy – Website Firewall Redefined

The Website Firewall was designed to give website owners peace of mind with a simple objective in mind; to keep your website safe by stopping the attacks from happening.

The logic behind the firewall is simple. It filters through all incoming website traffic and intelligently identifies good and bad traffic. All good traffic is allowed to hit your website and all bad traffic is blocked, which protects your website. In the end, the process looks a lot like this.

How the Sucuri Firewall Protects Websites

Latest Enhancements

The last major update to CloudProxy occurred in February, and back then, our update focused on a few key structural points:

  1. CDN Support (i.e., MaxCDN, CloudFlare, etc..)
  2. Reporting (i.e., Visualization)
  3. Point of Presence Expansion (i.e., More servers world wide)
  4. Back-end Rewrite (i.e., Code Refactoring)

In this update, we’ve focused more on the user experience, while still making some functional updates. Over the rest of the post, we’ll go over:

  1. Real-Time Monitoring
  2. An Improved Onboarding Process
  3. Country Blocks
  4. Enhanced Denial of Service (DOS) Protection


Read More

Desktop AVs and Website Security

Brian Dye tells the Wall Street Journal that antivirus tools like his company’s Norton suite are effectively “dead” because they catch less than half of all attacks, but from where we sit, that’s really just half the story.

Does Brian mean that antivirus defenses–also know as “AV”– are useless? Probably not. Just like you should get a flu shot to protect you from known viruses in the real word, you should also keep running AV to protect you from known viruses in the virtual world. We think a better way to put it is this; an AV alone isn’t enough to protect your computer because the websites you visit are constantly putting it at risk with new, unknown, viruses. When you look at websites, the same principle applies. Every day, we clean infected websites and webservers that already had some kind of antivirus or security software installed and we never tell our clients to just get rid of the security software.

The main reason that these sites get hacked is that the core of most security software relies heavily on signatures of known virus families. In the past, this worked well because there were just a few variants of viruses, which made it simple for research teams to study them and release signatures. However, the amount of malware now being created and released is so astronomical –because there is a lot of money in it– that a manual process is almost impossible. By the time researchers are able to dissect one malware string, a thousand more will already be released. So, even if you have an antivirus, your site and server can still be at risk. The key is to protect yourself from these bad outcomes before they can infect your site or your computer.

What can you do?

First, you have to be able to detect and respond to compromises quickly. Second, security, like winter weather, is all about layering. The more layers you have, the more comfortable you’re likely to be.


Read More

Case Study: Analyzing the Origins of a DDoS Attack

Recently a client was experiencing a massive layer 7 DDOS attack, generating tens of thousands of random HTTP requests per second to the server. The architecture of the website included a cluster of three web servers responsible for handling all incoming traffic, which did little to alleviate the pressures brought about the attack.

An interesting point about layer 7 DDOS attacks, aka HTTP flood attacks, is that they have little dependency on bandwidth allowing them to easily take down a server by overloading its resources. Depending on the web server and application stack, even a low number of requests per second can choke the application and backend databases. On average, attacks greater than 100 requests per second have the potential to bring down most mid-sized websites.

Anatomy of a Layer 7 DDOS Attack

This is exactly what the client was experiencing. The attacker was hitting non-existent URLs on his site and generating requests like this:

GET /music - 404 (not found)
GET /italian-wedding - 404 (not found)
GET /love/you - 404 (not found)
GET /bluechevy - 404 (not found)
.. and thousands more random words ..

The attacks were at very high speeds and coming from various sources around the world. Here is a map of the various connections. This occurred over a short time period (few hours):

ddos-map-2014-04

In total, we recorded a little over 29,000 unique IP addresses around the world. The US was the number one source, and below you’ll find a graph of the top ten countries associated with the attack:

Sucuri - Analyzing DDOS Attack

We were curious about the make up of the attack, specifically where it was coming from. To account for this, we leveraged the p0f tool (a tool to identify the operating system of the IP addresses attacking the site). This brought about a very interesting revelation:

Sucuri - Analysis of DDOS Attack Desktop Origins

What we found was that 85% of the incoming IP addresses were originating from desktops and not from web servers. Approximately 15% were using Linux, FreeBSD or were not identified. This, coupled with the fact that the IPs originating from cable / ADSL providers, allows us to deduce that the client was being attacked by a large desktop botnet.

Mitigating Layer 7 DDOS Attacks

The issue with this type of attack is that server-level caching is unable to stop it. The incoming URLs are dynamic and the application forces a reload of the content from the database for every new request that is not in cache, which creates a new page. Attackers know this, making it the preferred method of attack for today’s Layer 7 DDoS attacks.

Botnet-based DDoS attacks on the application layer limits resources, curtails revenue, and yields customer dissatisfaction, among others. DDoS attacks are among the most difficult problems to resolve online, especially, when the target is the Web server. – International Journal of Computer Appplications

To protect the client, we used our emergency DDOS protection feature, which uses JavaScript tricks to prevent malicious bots from hitting the site, while allowing access to valid users using real browsers. We combined that with our intelligent log correlation system, which allowed us to pinpoint the IP addresses and traffic pattern, blocking the incoming attack at the edge (a.k.a via the Sucuri Website Firewall) before it was able to overload the web server.

Are you Experiencing a Layer 7 Attack?

Have you been experiencing issues like what was described above? Do you have logs you can’t make sense of? If so, we’d love to see them. If you have logs to share please send them to us at soc@sucuri.net.

If you need help protecting against DDOS attacks, please don’t hesitate to let us know.

Patching The Heartbleed OpenSSL Vulnerability

Security Researchers have discovered a very serious vulnerability in the OpenSSL library that is used to power HTTPS on most websites. Many news sources are now covering the story, and we recommend reading their articles to understand the scope of what is happening and the impact of the threat:

To summarize: It is big. It allows an attacker to extract information that was supposed to be private, including SSL private keys themselves. ArsTechnica explains it well:

The bug, which has resided in production versions of OpenSSL for more than two years, could make it possible for people to recover the private encryption key at the heart of the digital certificates used to authenticate Internet servers and to encrypt data traveling between them and end users. Attacks leave no traces in server logs, so there’s no way of knowing if the bug has been actively exploited. Still, the risk is extraordinary, given the ability to disclose keys, passwords, and other credentials that could be used in future compromises.

The Tor team summarizes their recommendation by saying, “If you need strong anonymity or privacy on the Internet, you might want to stay away from the Internet entirely for the next few days while things settle.”.

What Should I do as a WebMaster?

If you own a website, you must do your part and patch your operating system. If it is a dedicated server, it is your responsibility. If you are on a shared hosting platform, contact your hosting provider to remind them to update their servers. To update your server with the patch follow these step by step directions:

1- Check if your site is vulnerable

We first recommend that you check your site on this page to see if it is vulnerable. If it is, keep reading to see what you need to do.

2a- Patching Ubuntu/Debian dedicated servers

If you run Ubuntu or Debian on a VPS or dedicated server, you will likely need to patch it yourself. A quick way to do that is by updating all packages on your operating system with the following command:

sudo apt-get update
sudo apt-get upgrade

Then restart Apache.

2b- Patching RedHat/CentOS/Fedora and most cPanel dedicated servers

If you run any RedHat-based server, you can patch your server by running:

yum update

Once all packages are updated, you should see inside /var/log/yum.log that OpenSSL was fixed:

# tail /var/log/yum.log |grep ssl
Apr 08 03:49:26 Updated: openssl-1.0.1e-16.el6_5.7.x86_64
Apr 08 03:49:27 Updated: openssl-devel-1.0.1e-16.el6_5.7.x86_64

Once that is done, you need to restart Apache for the fix to take effect.

2c- Other servers

If you are on a shared host, you can’t do anything. You’ll need to contact your hosting company and wait for them to run the patch for you.

If you are using any other Linux (or BSD) distribution on a dedicated server, you need to follow their steps to update OpenSSL.

3- Restart Apache

Do not forget to restart Apache (or Nginx). We are seeing many patched servers still vulnerable because they forgot this simple step.

4- Generate new certificates

This vulnerability was just disclosed a day ago, but it is possible that a malicious party has known about it for longer than that. If you run a popular web site or take confidential information, you might want to generate new certificates and encryption keys just to be on the safe side.

CloudProxy users Protected

If your site is behind our CloudProxy web site firewall, you are already protected against this and any exterior threat. Anyone can sign up for it, regardless of host or CMS and get their sites protected in just a few minutes.

Sucuri CloudProxy Website Firewall Improvements

If you are are a regular reader of our blog you probably know about our CloudProxy Website Firewall, it launched publicly a year ago. Since then, our team has been extremely focused on improving it, making it more effective and efficient for everyday website owners.

If you are not familiar with CloudProxy, I highly recommend reading some of the documentation and benefits of it:

In fact, if you have a website, why not try it out?

Read More

Layer 7 DDOS – Blocking HTTP Flood Attacks

There are many types of Distributed Denial of Service (DDOS) attacks that can affect and bring down a website, and they vary in complexity and size. The most well known attacks are the good old syn-flood, followed by the Layer 3/4 UDP and DNS amplification attacks.

Layer 7 DDOS

Today though, we’re going to spend a little time looking at Layer 7, or what we call an HTTP Flood Attack.

Read More