How well are credit unions prepared for data center outages?
August 30, 2007 – 12:31 pmBack in July, 365 Main, a major data center hosting companies such as c|net, Craigslist, e-surance, and the Oakland Raiders, lost power. For most of the day. And 365 Main has the time, money, and resources to make sure that (almost) never happens. Here is the full story straight from 365 Main. Jesse Robbins has a slightly more digestible version here at O’Reilly Radar.
So if a major provider like 365 Main can drop clients for half the day, what protections are in place in our CU’s? What are reasonable expectations from our members? Do we need to stay up for 24 hours after a power outage, or just survive until the power company comes out to replace the broken transformer or power pole? Obviously the most important thing is data loss and, for the most part, as long as the CU has the appropriate safeguards in place to protect against power surges, the servers and data should be safe from melting. After basic data preservation then comes up time. As a member, how would you feel if you went into a branch and the teller said that they couldn’t actually do anything on your account until tomorrow because their computer system was down? You probably wouldn’t have online banking to check your balance and the CU would probably have to limit the amount of cash they could withdraw for members since they don’t actually know how much each member has. So out comes the pen and paper to hand write which members took how much money. Not an ideal situation.
Put the members’ hat on. What would they expect? I’d be miffed if I couldn’t get cash out, but I’d probably survive. For a day.
2 Responses to “How well are credit unions prepared for data center outages?”
Robbie,
We’ve been wrestling with this question quite alot recently. Our main branch seems to lose power at the slightest thunderstorm. We’ve been going ’round and ’round with asking what do we really have to have in order to provide good service and maintain security? The answer is, we need full power! So we’ve also been getting quotes for generators. We’re incorporating these kind of issues into our disaster plan as sort of a “mini” or “short-term” disaster.
What we’ve found with our multiple power outages is that most members do understand and get over the miffededness quickly. Who hasn’t lost power at their house? If we were a big impersonal institution whose members didn’t really like us that much, then it would be much more of a problem. In saying that though, we still see clearly the need to make special arrangements to maintain the best possible service. There’s alot that can be done with a telephone line and a charged up laptop!
By Dan Veasey on Aug 30, 2007
I mentioned it in the comments of Jesse’s article and our guys have podcasted about it (with an IT security slant), but we can’t say it enough: Incident Response Plan.
(Say it with me: Incident Response Plan.)
The steps you take when something like this does happen, (and these things do happen) need to be documented and followed.
Do you run to Starbucks and email your members? (Would you even be able to get to their email addresses?)
Do you post something on your website? (Is your website even up?)
What physical services do you still provide even if the power is off? (Check cashing at all? Up to a certain amount?)
All these things need to documented AND FOLLOWED when something like this does happen. The best IRP in the world doesn’t do you any good if the power is out and it’s on someone’s computer.
PS. As I was writing this comment, the power went out in my apartment…eerie.
By Mark on Aug 30, 2007