What do you talk about when your power has been out for five of the last seven days and your Internet has been down for the last 24 hours. Disaster Recovery. We are fortunate for sure. But in today's day and age, the local Internet is as important as the roads we drive on.
Today in Health IT Disaster Recovery. My name is Bill Russell. I'm a former CIO for a 16 hospital system and creator of this week Health. A set of channels dedicated to keeping health IT staff current and engaged. We wanna thank our show sponsors who are investing in developing the next generation of health
Gorian Dynamics, Quill Health, Tow Site Nuance, Can Medical and Current Health. Check them out at this week, Health dot. Slash today. All right. Today, what are we gonna talk about today? Let's talk about disaster recovery. I am sitting outside on my back porch at six o'clock in the morning recording this show.
And the reason I am doing that is because we, live in Naples, Florida, and obviously everyone is aware of the hurricane that has hit and we were without power for about four or five days,
and we consider ourselves extremely fortunate. No damage to the house, no injuries to the family.
Given the, , number of, of deaths that Ian has caused,
really consider ourselves incredibly fortunate. So what I describe to people is that I have nuisance problems down here, but it it got me to thinking about disaster. Because we were at without power for a while. We have a generator and,
you know, when you have a generator, it, it really, being without power for that number of days didn't really impact us all that much.
We were getting down close to the end of the propane that runs our generator. , I think we were down to 30%, so we had probably about a day or two left on the generator before,
we would've been without power as well. But, ,
we, we got through that and oddly enough, the first two days people were texting me.
calling me and telling me what was going on down here. Cuz we really didn't know because we lost our cable and internet really early on in the process. Probably, I would say real close to when the storm was ending, everything went down and,
so you guys knew more than I did. You were watching it on tv, you were seeing the updates, you were hearing the stories, and we had no way of really doing.
internet and power were restored. Power went down again,
about 24 hours later. Went down again for about another five hours, but then they brought it back online. The, , internet now has been down for the last 24 hours, and I'm reminded of the story of, the person on, , on an airplane getting.
Just really upset about the fact that the internet wasn't working on the airplane and the person who was telling the story was just saying, , do you realize how amazing it is? What's going on right now? We are traveling from New York to LA in, , five and a half hours and , this is something that people used to have to do. , wagon train and they risk their lives and people died on the journey and that kind of stuff. And it's just such a miracle that we can go from New York City to LA in five hours and be sitting in a meeting or, or doing whatever, , in that timeframe.
And yet we are complaining about the fact that we don't have internet. Um, I now find myself on the side of the person who was complaining that they didn't have internet. , the world has changed. The world's changed after the pandemic through the pandemic. My job relies exclusively on the internet. It's how I communicate with all the of my employees, my staff around the country and literally around the world.
Um, one of my staff members right now is in Australia for a wedding, and she is working while she's down there for the month. And, , so I rely on it for communicating with the. I rely on it for just about everything, quite frankly. I mean, you're listening to this as a result of the internet, which I am now, , sitting outside, you probably hear crickets cuz I'm sitting outside on my back porch cuz it's the only place that my cell phone really works and I'm tethered to my cell phone.
I'm recording this and I'm gonna upload this to the, , podcast channel. And it got me thinking about. From a de disaster recovery standpoint, are we thinking about remote workers and the impact on remote workers? I would imagine right now, , you know, I've heard a couple of the stories of the health systems that have been impacted in southern Florida.
I, I heard of a hospital who, that their roof came off. I heard of a CIO who, , whose home was, , , at least the first floor was, was completely flooded. And their, , equipment ended up strewn all over the yard. , I heard of, urgent care clinic. That were flooded out in, , key areas as well. And, it just great, phenomenal work.
Lee Health did phenomenal work during this. They, they took on, , a lot of patients. There was a lot of transports as you would imagine, from , nursing homes and that work. Went off without a hitch, so I'm not gonna be talking about it from that perspective. I'm sure that those teams had disaster recovery plans in place and they implemented those disaster recovery plans.
What I'm going to challenge you on today is what's your disaster recovery plan for the local internet, for your staff that is working remotely? What if they're without internet for an extended period of time, and how much of your staff. , I don't know, aggregated in a small area like let's say Lee County in.
In, , Florida, cuz Lee County and Florida is still, I think 50% without power. I think we are down to, in fact, the Florida Power and Light did a phenomenal job. The, , all the, , outlying, , counties, which we are considered have almost been 95 to 98% restored. It's the, the, the path of the eye. , went through Lee County and a couple other counties, but even those are well over 50% restored at this point.
And there's some places that they can't restore. But getting back to my point, getting back to your, the, the employees, what's, they're aggregated in an area that's hit by a disaster. Do you have a plan around those things now as CIO in Southern California? Took disaster recovery very seriously.
We did a lot of tabletop exercises. We tried to consider all the different scenarios. We brought in consultants to talk to them about scenarios. In fact, we moved our data centers out of Southern California. We moved them, , to the cloud, and we also moved them to a co-location facility out of, , Las Vegas.
When we did move to Las Vegas, we did, , analysis of what the major, disasters were that Las Vegas. , be inundated with. And it turns out the biggest disaster that, , Las Vegas needs to be prepared for is a disaster in Southern California because all of those people will end up flooding the first city they're gonna run into as they leave California is going to be Las Vegas.
And so that's the biggest disaster you need to be worried about. The second biggest is the, , rail lines and the rail lines, , can carry, , nuclear, , waste. to facilities that is storing nuclear waste in Nevada. So you have to be worried about those things. , so you have to do that extensive analysis to determine what the impact is going to be on your health system.
If, because that was our core data center, if that core data center went down, that would impact all of our hospitals. . Now we had backup data centers as well. So this is sort of rambling.
That's what you get at six o'clock in the morning out on my back porch. You get rambling about disaster recovery. I, I think it's an important topic. I think it needs to be revisited every year at a minimum. , maybe every six months is, , probably more appropriate. I firm believer in the tabletop exercises.
Your person running the tabletop exercise cannot be creative enough. , you know, I hear the stories over the years, , Tulane University in, , New Orleans, , during Katrina, they could not get back into the building, back into the data center.
They couldn't find backups. They couldn't run payroll. And so you can't be creative enough with the disasters. And when people say, Oh, that would never happen, , you don't know what would happen. And so, play those out. See what happens. Even to the point of saying, Okay, the CIO has died.
Who's making the decisions? What's the decision making matrix within the, , organization? Because those are possibilities in a disaster. So, Thought I'd, , update my disaster recovery, , message to y'all while I am living through this, disaster that has hit southern Florida.
And, um, again, I wanna thank everybody for reaching out. And things are, as I said, we have nuisances right now, but, , really, , our fortunate we have no property damage and, The family is fine. So consider ourselves, , blessed and lucky. That's all for today. If you know of someone that might benefit from our channel, please forward them a note.
They can subscribe on our website this week, health.com or wherever you listen to podcast. Apple, Google Overcast, Spotify, Stitcher, which all rely on the internet. You get the picture? We are every. We wanna thank our channel sponsors who are investing in our mission to develop the next generation of health leaders, Guardian Dynamics, Quill Health, to site nuance, Canon medical, and 📍 current health.
Check them out at this week, health.com/today. Thanks for listening. That's all for now.