A couple of days ago we were busy upgrading software across the fleet. A pretty regular task, and so it was with some concern that half way through the work the phones lit up and public access to our network became impossible. We’ve been DoS’d before. Denial of Service attacks are pretty common place and are usually levelled at one of our web hosting customers’ websites in an attempt to knock them offline. We’ve been dealing with them for some time. But like any defence system it has to fit the anticipated threat to be economical. On this day, we were hit by a tsunami that overwhelmed not only our internal defences, but also those of our data centre and upstream providers.
That was around 48hrs ago – we’re all pretty tired, but at the same time we are excited, and in many ways we are grateful for the challenge that was placed before us. The suffering we first felt was quickly transformed into a determination to improve our situation. As Zen master Thich Nhat Hanh says – No Mud, No Lotus. Sometimes we have to experience a good kicking in order to reveal our greatest qualities – that being perseverance and the desire to provide an awesome hosting service. So, today we have altered our network infrastructure to cater for attacks far greater than those that have caused us and our customers so much pain these past few hours. Here’s what our CEO, Simon Blackler had to write to customers today:
Krystal has been attacked 5 times in the last 48 hours. These DDOS (Distributed Denial of Service) attacks have been the most concerted, prolonged and varied we’ve seen in 12 years in business. The sheer size of the attacks exceeded the capabilities of the equipment and systems we had in place to mitigate them. They even caused problems for our datacentre and upstream Internet providers.
As a result at certain points users couldn’t reach our servers. I’d like to apologise personally for any inconvenience caused.
3 different vectors used (often at the same time);
UDP/DNS Amplification (Port 53)
TCP Ack Flood
UDP/NTP Amplification (Port 123)
Highest throughput – 2.2 Million packets per second
Largest attack – 720.3 Million packets (this wasn’t in isolation)
Biggest “botnet” (network of compromised machines) – 235,681 hosts
Duration – 3 attacks were 1 hour long or more (Typical DDOS attacks last 5-15 minutes)
I’m writing to let you know that we’re NOT going to be taken offline. I’ve spent 12 years building this business from scratch and I’m not going to let someone who can’t create something of their own try and destroy it.
So, here’s what we’ve done…
Throughout yesterday (Wednesday 22nd July) I re-architected our network to allow us to pick up an in-line DDOS mitigation service. In the early hours of this morning we exchanged contracts and as of 10 minutes ago we are running with that protection in place. This new service automatically mitigates attacks, often in less than 60 seconds. Critically, in the largest of attacks only the target server will go offline, not the whole network.
While there will no doubt be some intermittent disruption while we fine-tune the solution it will be negligible compared to a full outage, and something we can schedule to minimise impact. I would appreciate your continued patience and support while we bed the new system in. It’ll be worth it.
Krystal have purchased this DDOS protection and are putting it in front of your website(s) at no extra cost because it’s the right thing to do. Because we appreciate that being offline is frustrating and can cause loss of business and reputation. And because we want you to feel good when you recommend Krystal to people you know, confident in the knowledge that we truly care about our clients. And we do. Every member of staff here went above and beyond yesterday, answering calls, tickets, Facebook and tweets and keeping the Status Blog updated, well into the night. I would like to commend them for their professional and friendly response to what was a stressful situation. (and a big thank you to all the clients that showed their support, it meant a great deal to the staff)
If there’s a silver lining it’s that Krystal just got even better. Thank you for your continued support and patronage.
We thank our customers from the bottom of our hearts for their patience and the overwhelming support they have shown to us throughout this difficult period. We look forward to continuing our wonderful relationship for many years.