As many of you with Slices in our Rackspace DFW Datacenter are painfully aware, there was a power outage in the early hours of Tue, 3rd Nov.
Firstly, it goes without saying that I personally apologise for the outage and I want to explain what happened. Although this will not change your experience, I hope it will go some way to easing concerns about a repetition of last night.
Secondly, it has taken much of today for me to get accurate information. For those that know me, you know I am a straight talker (to say the least) and I wanted facts before I approached you with details.
So, what happened?
At around 12:29am CST this morning, the DFW data center experienced a power outage during a routine (non impacting) maintenance. Clearly, this non-impacting maintenance became impacting and many Slicehost customers in the DFW DC experienced downtime.
Power was restored within five minutes and most Slices were up and running in a good timeframe. However, some of you did experience lengthy delays while we restored your Slice. We had the Systems team working from the moment we knew something had occurred and they did not stop until everyone was up and running again.
The issue was further complicated by internal DNS issues which were caused by the unexpected power outage.
However, this does bring up a couple of specific points I would like to discuss:
We did not post a notice about this maintenance. Until now, we have restricted notification of maintenance windows to those that might, under normal circumstances, have an impact. We didn't expect this period to be impacting and so we did not post a notice.
This turned out to be very wrong and, as a result of this, I will be posting more routine maintenance warnings on our status blog.
The second point is that I feel our communication lacked in a couple of areas. We did not keep you informed as regularly as we should have done. Even if there was no specific news, I feel regular updates are essential so you know we are dealing with the issue. I have already changed our procedures and there is a clearer, more defined, route for communication.
I make no excuses for what happened. It reflects on us badly and it affected you negatively. This is not something I find acceptable and I will continue to work to provide the best service for you.
I know you will be concerned and if you have any questions then please email us or come to our chatroom.
Paul