Late last week one of our co-located customers began experiencing a DDoS attack targeting two of their sites. At it’s peak the traffic hitting the server was around 400mbit/s, with thousands of requests coming in per second. While not the largest attack in history, the payload consisted of an 8 kilobyte random string sent directly the the web service (no GET request). This is one of the worst things you can do to Apache, as it fails miserably on it’s own at handling high volumes of requests like this. It considers it valid, so it sends a 414 response (URI too long) back to the client. This gets pretty messy pretty fast.
This attack was quickly overwhelming the server. Once the responsibility of getting rid of it was handed off to me and I was granted access to the server I was able to diagnose things fairly quickly. The server was running a current LTS release of Ubuntu and it was up to date which was good to see, but it wasn’t tuned. Keeping a server up to date is one thing, but configuring boot and sysctl tunables is just as important. It’s a trivial task to accomplish and depending on the workload the benefits can be exponential.
Anyway, I got the server properly configured so it was able to deal with all the connections and I was able to pretty easily fingerprint the attack. I set up a script to pipe the ips that were sending these bogus requests into iptables which worked for a while but every time a new onslaught of ips were thrown into the mix, Apache would hit it’s connection limit before they could be added to the firewall. Enter Varnish.
I’ve been experimenting with Varnish for a while, using it as cache/page accelerator. I’d never attempted to use it as part of a DDoS mitigation solution but suspected it would help. How well it would help I didn’t know until I got it set up in front of the sites under attack. It should be called “Vanish” cache, because from the moment I put it in front of Apache the attack disappeared. Varnish immediately drops the request because it’s bogus, so Apache never sees it. With a little bit of tweaking to Varnish’s configuration I was able able to improve the load time of their site even while getting DDoSed.
So not only did Varnish stop the attack dead in it’s tracks, I was able to get a fair bit of the requests into the Varnish cache so it would serve the request directly. Their sites load faster than they did before I put Varnish in front of Apache, and the DDoS attack is still ongoing as I type this. That’s some pretty cool shit if you ask me.