> I've seen these attacks mitigated single-handedly by an experienced fellow with no access to fancy equipment. I believe it went something like this, change the DNS to point at some EC2 instances to do front-end load balancing with some scripts that detect and drop connections from attacking IP addresses and do severe rate limiting. You can proxy legitimate requests back to the original servers.
That isn't how it works, but I appreciate you trying to explain it to me (I have personal experience as a former employee of a hosting company, in this exact position). A typical DoS attack is designed to flood the pipe, usually with UDP. UDP spam attacks are far simpler than layer 7 attacks that you describe, and layer 7 attacks (if small enough) are easier to handle.
As a point of reference, I don't even have access to a botnet these days but I could knock your home cable connection offline in about five minutes of my time.
In your scenario, if you were to update DNS to point the victim at some EC2 instances, you'd accomplish DoS attacking EC2 instead as they only offer you gigabit connectivity on most EC2 instances. If the DoS attack is multiple gigabit, that isn't going to help you unless Amazon steps in and mitigates about as best they can. A DoS mitigation strategy generally has nothing to do with your application. If an engineer's first answer to a huge DoS attack -- as in, larger than the uplink -- is iptables or "scripts to detect and drop connections", that engineer is uninformed. I'm sorry.
I have personally witnessed a 40 gigabit/sec attack. I'm sure the smart engineers at CloudFlare have seen bigger. Upstreams don't give a shit what's flowing across the wire, and a small TCP SYN flood directed at your server won't gain any attention from your host or their upstream. A multiple-gigabit attack that takes down the entire uplink will.
Depending on your position, there really isn't one. The hosting company can implement one, which they might be able to insert in your path if you are at the receiving end of an attack. There are Cisco products and a bunch of up-and-comers that can do this.
If you're a "typical startup" with an Amazon footprint, you have no mitigation strategy for flooding attacks aside from not attracting them. If someone points multiple gigabit at you, there is just about nothing you can do except hope Amazon can do something.
There's a range[2] of ISPs who will sell DDoS protection to you, either as an addon when you host with them, or as an external service (re-routing your traffic).
E.g. StormOnDemand just recently added it to their portfolio[1], which is note-worthy because they actually list prices right on the website.
Either way, even without "explicit protection" any ISP beyond mom&pop-size deals with these attacks every day and will sort them out for you for free the first couple times. Only when they turn into a habit or become so huge that they have to talk to their upstream they will politely ask you to throw some money their way.
Pulling the plug immediately is definitely not normal. However considering Pastie was apparently a sponsored account it's at least somewhat understandable (albeit a terrible PR move).
For serious accounts (in the 6 digits/year) absolutely not, unless the attack is large enough to affect other customers.
Admittedly RailsMachine looks very small, in all likelihood their pipe was rather easily clogged and they simply didn't have the choices that larger ISPs have.
> For serious accounts (in the 6 digits/year) absolutely not, unless the attack is large enough to affect other customers.
If it doesn't affect other customers, a hosting company won't act or even be aware, in most cases. They'll just send you a bill for the transfer. If someone attacks you and it impacts other customers, you get nulled. I'm aware of 7 digits/year and 8 digits/year accounts through industry anecdotes that have had machines nulled. The engineer operating the null doesn't say, "oh, that's X, maybe I shouldn't fix the network for my other customers".
There's a bit of middle ground between "sending a bill" and nulling.
I've been hit by two larger attacks in the past (GBit/s range) and the respective ISPs were both extremely supportive, switching our IPs while they tightened their filters. Neither billed us a dime despite our ingress spike making quite a bump in their charts and a lot of handholding over 2-3 days.
I understood the original comment about EC2 as a mitigation strategy so the hosting company's infrastructure doesn't receive the brunt of the DDoS gigabits (EC2 will) and it can still service all their other clients. In that case, even if the DDoS'd site will still be flooded and unavailable, its hosting plan doesn't need to be canceled outright. I suppose they could just skip EC2 and null-terminate the DNS until the DDoS stops, but I don't know if this has further implications.
In any case, thank you for a very informative post.
That isn't how it works, but I appreciate you trying to explain it to me (I have personal experience as a former employee of a hosting company, in this exact position). A typical DoS attack is designed to flood the pipe, usually with UDP. UDP spam attacks are far simpler than layer 7 attacks that you describe, and layer 7 attacks (if small enough) are easier to handle.
As a point of reference, I don't even have access to a botnet these days but I could knock your home cable connection offline in about five minutes of my time.
In your scenario, if you were to update DNS to point the victim at some EC2 instances, you'd accomplish DoS attacking EC2 instead as they only offer you gigabit connectivity on most EC2 instances. If the DoS attack is multiple gigabit, that isn't going to help you unless Amazon steps in and mitigates about as best they can. A DoS mitigation strategy generally has nothing to do with your application. If an engineer's first answer to a huge DoS attack -- as in, larger than the uplink -- is iptables or "scripts to detect and drop connections", that engineer is uninformed. I'm sorry.
I have personally witnessed a 40 gigabit/sec attack. I'm sure the smart engineers at CloudFlare have seen bigger. Upstreams don't give a shit what's flowing across the wire, and a small TCP SYN flood directed at your server won't gain any attention from your host or their upstream. A multiple-gigabit attack that takes down the entire uplink will.