Sorry for the delay in providing this report regarding the last network issue
Date: October 7th 2007
Timeline: 02:15 – 03:35am (Irish time)
Affected Customers: Any customer on the shared firewall that has a dedicated server or has colocation with us was affected during this incident.
This also included our shared hosting clients.
What happened?
At around 2:15am on Sunday 7th of October a segment of our main network was sluggish and people would have experienced latency and packet loss.
Why?
As you may know our main network is firewalled. We have a pair of firewalls setup in HA (high availability) to protect the bulk of our clients, which includes all our shared hosting clients on both windows and linux, as well as a large number of clients on dedicated servers or with colocated machines.
Similar to the events of September 11 this year this was mostly because the firewalls we’re using have 100meg ports and as such are easily flooded by this simple attacks. We’ve already put the wheels in motion to upgrade these and we hope to announce the upgrade at the end of this week.
A brief timeline of events is shown below.
02:15: Alerted that sites on the network are not reachable.
02:20: A check reveals that any site behind the shared firewall is
not accessible.
02:30: A reboot of the firewalls is not successful in getting a
response. so an engineer is dispatched to Dublin
03:30: A check by the Engineer on site indicate that one customer
box is sending out masses of UDP traffic. The firewall is attempting to stop all the traffic at the cost of bring down everything else includingthe local console.
03:35: The customer port is disabled and the firewall becomes responsive
again.
About the Author: Michele Neylon
4 Comments
Comments are closed.
Would a customer normally be notified if this happened?
If what happened?
have to say it was spotted and responded to very quickly.
I was awake and witnessed it, but the team were on hand to fix. I fell asleep and woke up at 6.55 to the start of the Chinese F1 Grand Prix and all was well. now I must subscribe to this here feed.
Brian – we try to be as responsive as possible, though our focus will always be primarily on the problem at hand. With regard to incident reports – I’d prefer to wait 48 hours and provide a proper one, rather than throwing something together
Michele