September 16, 2020: What challenges are we facing in data protection today? Josh Peacock from Sirius, Michael Friesen from Cone Health and Zak Pellecchia from Rubrik join us to talk about Rubrik’s revolutionary data control system. Say goodbye forever to massive tape robots in your data center that have to be fed and cared for at night. How has this environment been simplified? What about adhering to backup regulations? Automation? How fast is operational data store growth right now? And what does the future hold for running the EHR in the cloud?
Rubrik Solution Showcase with Josh Peacock, Michael Friesen & Zak Pellecchia
Episode 304: Transcript - September 16, 2020
This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.
[00:00:00] Bill Russell: Before we get started. I want to share with you something that we are extremely excited about here at This Week in Health IT and that is clip notes. Clip notes is the fastest growing email lists that we've ever put together. if you can't listen to every show, but you want to know who was on and what was said, the best thing to do is to sign up for clip notes.
One paragraph summary, key moments in bullet point format with timestamps and one to four video clips from the show. It's a great way for you to stay current, share insights with your team and maintain your commitment to their development. During these [00:00:30] extraordinary times, the best way to sign up. The easiest way to sign up is just send an email to clip notes.
CLI P N O T E S. @ this week in health it.com and it'll kick off an automated workflow. You'll get an email back from me, click on that link and you are off to the races. so don't delay send that email, get signed up today now onto the show.
Welcome to This Week in Health IT. Today we do a solution showcase focused on data [00:01:00] control and data protection. My name is Bill Russell, healthcare, CIO, coach, and creator of This Week in Health IT a set of podcast videos and collaboration events dedicated to developing the next generation of health leaders.
This episode, every episode, since we started the COVID-19 series has been sponsored by Siriushealthcare. Now we're exiting that series and Sirius has stepped up to be a weekly sponsor of the show through the end of the year. Special thanks to Sirius for supporting the show's efforts during the crisis and beyond.
I enjoy doing solution showcase episodes for two [00:01:30] reasons. Primarily the first is these organizations are contributing financially to the production of our program and our mission to develop the next generation of health leaders. And also because it gives me an opportunity to explore these solutions, these really interesting solutions that I believe can have an impact on our health systems and on the care in our community This shows dives deep into data protection and data control with guests from Cone Health, our sponsor Sirius Healthcare and Rubrick, a Gartner magic quadrant [00:02:00] leader in this space. Here's our solution showcase. I hope you enjoy.
All right, today, we're going to look at a really special topic that I've wanted to look at for a while now and that is data protection. Especially with all the stuff that's going on with ransomware and disaster recovery change of things. So I'm really looking forward to this conversation. We have three distinguished panelists or distinguished or not. We'll find out in a minute. We have Josh Peacock, former guest, Healthcare Solutions Architect with Sirius, Good morning Josh.
[00:02:30] Josh Peacock: Morning
Bill Russell: Michael Friesen, Senior Technical Specialist with Cone Health. Hi, Michael.
Michael Friesen: Good morning.
Bill Russell: Good to have you. And then, although Michael is a true security operations guy, he does not have the, the video. It's all taped up and whatnot. We'll superimpose a picture there for him. And then, Zak, I didn't ask you how to say your last name and I should have Pellecchia. Is that right?
Zak Pellecchia: Yeah, yeah.
Bill Russell: Got it. Data Protection Specialist with Rubrik [00:03:00] and Rubrik is a Gartner leader in the multicloud data control space. Welcome to the show. I should ask people how to say their names before I start the show. I don't know why I keep making that mistake.
You would think this is episode like number five, but this is actually going to be episode like three oh five or something like that. So crazy. This is a topic that keeps coming up and in itself in different ways. And a lot of times it's through challenges or [00:03:30] data loss, or ransomware or different things to that effect. And I wanted to dive into this. And I really appreciate you guys coming on the show. So let's start with the challenge and I'm not going to call on any of you. I'm just going to throw stuff out and then whoever wants to take the question, feel free to go after it. Let's start with the challenge, what challenges are we facing in the area of data control today?
Josh Peacock: I can answer a few things that we're seeing [00:04:00] particularly in my experience around some of our Epic clients and some of our other, just general healthcare clients as well, too. We're seeing some significant and just almost uncontrolled growth in data, which I guess we've heard about that for a number of years.
But even in, that had been traditionally like unstructured, even instruct sure. Databases are growing to these massive sizes that we're trying to back up. Ransomware, even though, and we'll dive into that a little bit further, but. For every one that we hear about, there's a client, somebody that's [00:04:30] getting hit by it daily.
That's recovering from things too as well. So that continues to just be, an area that gets, really beat on in healthcare. as a target, they're becoming more sophisticated than that as well. so I don't know if Michael or, Zach, if you got anything else to add there too, but those are two of the big ones that I see.
Bill Russell: Michael, I'm going to, I'm going to kick it over to you. The, so you guys, you're operational, you're in the role right now. What kind of things are you seeing at Cone Health that causes you to look at this space and to look at, [00:05:00] a solution like Rubrik.
Michael Friesen: For us it came into play when we were trying to get away from some of the legacy complex, backup solutions. the automation opportunities with the rest API, were huge for us. that goes into. The data protection, because, we automate setting up all of the backups and verifying that they're there. we, if we need to recover, data [00:05:30] there's still some rest API and automation opportunities for us.
the data growth, as you mentioned, it's not just having one. Database, that's 30 terabytes. It's five copies of that are all slightly different. And so those challenges certainly exist for us as well.
Bill Russell: so is that a window problem? So back in the day back, back when I was CIO, we had these growing data stores and our windows were closing.
Like we couldn't back it up between the windows we needed to [00:06:00] back it up. Is that still a problem or is that an old problem?
Michael Friesen: It's still a concern. and it's a concern that you still have to work around to some extent, and obviously getting rid of tape, I don't want to look at a backup solution that still mentions tape. that helped tremendously, being able to tweak some of the threads and having a, a distributed node based system like Rubrik has helped us get [00:06:30] those backup windows down.
Bill Russell: Yeah. Here's a question I've I read about these ransomware things.
So I think UCSF was the most recent one. And, actually, Zak I think I call on you for this one. When I look at ransomware, we see the health system get attacked. They pay the ransom and I'm always like, I'm always sitting there going, I want to talk to somebody cause I want to find out like, didn't you have a backup?
Why. how did the ransomware, how were they successful? I understand how they can potentially get [00:07:00] in there and lock it a certain data set down. Couldn't I just wipe out that data store restore it mean help me to understand how ransomware works. how are they successful?
Zak Pellecchia: Yeah. I think it's, like Josh said that they're becoming more and more intelligent. Like they're figuring out that people are just restoring and not paying the ransom. So they said, what can we do to prevent people from being able to restore? And the first thing they did is start targeting. Common backup file extensions. there's a lot of, proprietary [00:07:30] extensions people use for backup files.
So the ransomware actually scan the file system, find all those files first, encrypt those files first and then move on to other files. So it's pretty impressive on how intelligent things are gaining.
Bill Russell: All right. so give me an idea since you're a, you're a data protection specialist, you're working with Rubrik. What do, what does Rubrik do? What do advance systems do to protect against that kind of a attack?
Zak Pellecchia: Sure. So maybe at the end of the day, I always ask the [00:08:00] question, if you're not, if you don't think that your production NFS or SMB shares are going to be protected in a ransomware attack, then why would you put your backup data there?
And that if you look around the industry, that's really what most of the competition does with Rubrik. we just built it. With security in mind. And in order to write to our file system, you first have to authenticate through our software. We use secure RPC calls to get to our file system.
So there's no IP address on the network where you can access [00:08:30] our file system, no NFS, no SMB, none of that. And if you're able to basically fake your way up through all that, which, Oh, by the way, we also have a zero trust methodology. So we make you reauthenticate each time, if you get to the file system, it's also a pen only, or immutable, meaning you can only add change files, you can't delete or modify any of the data in the actual file system.
Just pretty, I think we're the only one on the market that has something like that.
[00:09:00] Bill Russell: Yeah. that's, it would take a, an extremely sophisticated attack to go after that. And what we found is when you have all those kinds of barriers, People just go find somebody that doesn't have those barriers and that's they just go to the path of easiest attack and the path to easiest, money. So that's really what they're after. At the end of the day, I want to hit on a handful of things. I want to come back to that operational data store growth. I also want to talk about cloud and I want to talk about simplifying the [00:09:30] environment, I think are the three things which we talked earlier prior to this, and those were the three things that sort of jumped out at me that I, that are, interesting. And so Josh, you work with a lot of clients and a lot of different clients. Give us an idea you've mentioned it in the intro that the operational data stores and , the data stores just in general are growing. Can you give us some picture of how fast they're growing and what you're seeing.
Josh Peacock: Yeah, sure. So we have clients that are [00:10:00] actively in the, let's say 40 terabyte range for their core databases in the Epic space. And with that, those are, expected to grow to somewhere around 80 to 90 terabytes in the next three years.
So that's pretty substantial growth. and then alongside that, we're seeing growth in, the reporting side of the world on Epic clients. And then we're also seeing, the blob storage or data management systems as well. So if that's on base or ed code or whatever ones you have there too, We're seeing [00:10:30] significant growth within those spaces as well. so not just the databases themselves, but then those, file storage locations that are ancillary to the core database.
Bill Russell: so you have the court database, you have the analytics and then you have the. the document store, the OnBase kind of things.
Josh Peacock: And are any of those growing at different pace? is the document store growing faster than the others? That's pretty organizational specific. I think especially if they maybe had just converted to [00:11:00] the new EMR or brought in the new document management system, if they're doing a lot of scanning of old paper charts and bringing those into that DMS, we see significant growth in those. Otherwise it's more of a, a steady linear growth on those.
Bill Russell: Do we face any specific challenges in backing up Epic?
Josh Peacock: So for the most part, the clarity databases are there. You're going to be some form of Oracle, whether it's exit data, Oracle on something or Microsoft SQL, and there's good tools [00:11:30] around how to support those large databases and back them up in a good clean manner.
The other items, when you start talking about. cache or is he translates into Iris. It's not, any fault. It's just the regular backups of those are, they don't run into them as often. So vendors don't build specific plugins and things like that to make those work, in the best fashion.
So we go back to a traditional, effort of sweeping files in, is it a way to accomplish those backups. And that's where we really run into, [00:12:00] concerns about timing. It can take a while for large multi terabytes files to be backup every day. and so that's where I think, some of the work we've been doing is to document best practices around that with the Rubrik team and some of our, and then, provide white papers out to the industry so that they can do it in the most efficient fashion as possible with what we have to work with.
Bill Russell: so back in the day, again, I'm dating myself here. we did snapshots, we did full backups. We did snapshots. We did [00:12:30] incrementals. We did. there's a lot of different things. Is the methodology still the same or has it changed?
Josh Peacock: Yeah. So we built a kind of a complete strategy for the organization around protecting their data. We start with physical snapshots on arrays, and then we take those and put those into a, like a Rubrik solution, for a longer term and more protection in case something would happen to that array or be effected in some way.
but yeah, absolutely. we tend to do falls and then followed by [00:13:00] incrementals after that, the operational database for Epic, which is the cachet slash Iris. traditionally hasn't played well with, incrementals. And that is one area that we see Rubrik shine though, is the. Incrementals for their solution, actually do a really good job on those, those.dot files that are make up the database.
Bill Russell: Josh, I'm going to go to Michael here in a minute, but, last question for you. I would assume, if you're going to do ransomware, Epic would be something or the EHR would be something. [00:13:30] Have we seen successful attacks on an Epic system? Is that something that they do target?
Josh Peacock: I haven't seen anything at this point. I think the, if there was something that was most applicable to a lot of those tax, it would be the backups, which, exactly explain how to protect some of that. and then there are those file shares that are associated with document management, either the Epic web blob or your OnBase document stores tend to probably be the most, targeted type of data, just cause it sits on a [00:14:00] sambar super shares.
Bill Russell: Alright. No, that's helpful. Michael, I want to go to you on two things. I do want to talk about cloud, but I'll get to that later, but I want to talk about ease of use. I remember these systems not being fairly complex, hard to manage. You mentioned getting rid of tape. That was, the first thing is we have these.
Massive tape robots in our data center that had to be fed and, cared for at night kind of stuff. So that's one level of complexity that goes [00:14:30] away. How else have we simplified these environments are? How has, have you simplified this environment at Cone Health, with the adoption of the rubric platform?
Michael Friesen: One of the strategies when we were looking at this was trying to get away from the traditional complex backup system where you actually had one or two backup specialists and they knew that product. And there's not necessarily anything wrong with that, [00:15:00] but you can certainly imagine if you've got a new system that comes in and, or an emergency restore of some sort, and you've got to find the backup guy, it can be a challenge and it delays our patient care.
we can't restore a system or restore patient data in a timely manner. If nobody can just jump in and figure out how this system works. so that was something that really struck us with the Rubrik system was the ease of use. but [00:15:30] I actually had one of the guys on my, I told him what the URL was.
He logged in. He was able to set up a backup for a new system in five minutes. So that allowed us to say, there's no reason when we get a very high profile, restore ticket or something like that to come in that, the one on my team, we have the team of seven for the data center that includes backups. [00:16:00] Everybody can do a restore.
Bill Russell: No, that's interesting. one of the things I'm measuring things against now, and we didn't talk about this ahead of time. So this is a curve ball for you guys. But one of the things I like about the system that I used for the, now I have a small business here, So I go out and I use loud, systems, I stand up, compute storage, and those kinds of things.
And, but yeah, I can just bake the backup right in, right as I'm provisioning the. The, server, I just click a few handful of things [00:16:30] and I set up, a backup structure. I could choose snapshots. I could choose incrementals and those kinds of things as I'm provisioning the system. So it has a level of automation and I assume that's all accessible.
We have APIs. Is that something that Rubrik can do, Zak, I guess I'll ask you or Michael, is that something a rubric can do.
Michael Friesen: I can tell you, I don't want to jump in front of sec, so I'm going to be quiet now. no, please come excited about piece. This for us was huge. We provisioned and [00:17:00] decommission all of our servers using scripts. We have dabbled with some automation tools, but at the moment for most of our windows servers, it's all PowerShell scripts. So we are setting up the backup when we deploy the server using the power shell. Another piece of that power shell script sets up, the backup, puts it in the right SLA policy then.
And I think this is just as important because one of the things I love to [00:17:30] see is servers go away. And so part of the decommission process, we go out check that there was a recent backup in the rubric system using the rest API. If there was, maybe we. We take another snapshot if it's been more than a day and we just put that in an archive SLA so that we've got it.
Otherwise we just retire those backups and it's done. And basically all we're doing is executing some PowerShell scripts that reach out [00:18:00] to these pieces and notably one of them being the rubric rest API to get all of this accomplished. So it's simplified our. Tasks around building and destroying servers tremendously.
Bill Russell: Michael, do you have a cloud deployment as well?
Michael Friesen: Yes.
Bill Russell: So how are you looking at backups and data control across, a multicloud environment or on prem and cloud? How are you thinking about that and how are you [00:18:30] using tools around that?
Michael Friesen: Who were, I would say we're getting started with some of that.
We've been using the cloud now, for several years, but it's continuing to grow and we're still continuing to understand and learn how to make the best use of that. That was something that was a very big deal to us about Rubrik, as far as the kind of. They seem to be forward thinking about the cloud [00:19:00] from the beginning as well.
So I have a cloud cluster in Rubrik or in our Azure cloud. I'm using that to back up some file systems that in the past had a Azure backup limit. There was a four terabyte disk backup limit. We didn't have that limitation using the rubric product in the cloud. so some of those things have changed and that's not a true any more.
We can backup [00:19:30] bigger file systems, but still having it all in one place to manage the cloud backups and the on prem backups so that we. We just have one place to go to interact with that is a big strategy for us because we're branching out into AWS. We're using Azure.
We've got on prem data centers. I don't think that it's going to get simpler from that standpoint. So as long as we can have one interface for all of those things, [00:20:00] for our backups and our data protection, that kind of helps keep it a little bit simpler than it could be.
Bill Russell: I think you're so far ahead of the game. And the reason is because you guys are already thinking in terms of automation and scripting. And that's one of the huge benefits of the cloud. And if you can script your own data center, your on prem data center, then, I think the sky's the limit. It can be interesting. Zak, you've gotten off pretty easy on this.
It's actually nice when you have a, you have a [00:20:30] partner talking about your solution, you have a client talking about your solution, but I'm going to bring it back to you. A lot of data control backups specs come from really the regulatory requirements that we have. what kind of, What kind of things does Rubrik or a system, a data control system have that helps us to adhere to those regulations or even automate some of that? Some of that compliance.
Zak Pellecchia: Sure. so there's quite a few different things that manage like retention, that are pretty interesting, but they're really the [00:21:00] two key drivers that I see from Rubrik are, Are the ability to detect ransomware, is one, which doesn't really fit the footprint of what you were talking about, but basically just detecting ransomware and being able to give you a list of all the effected files and that you restore.
so just giving you more control over your data. And then the second is, an integration where we actually do data discovery. And we can actually using backup data, without running scans on your production system with no agents, we can actually [00:21:30] use the backup data as we're ingesting it and scan it on the backup system for things like HIPAA, PII data, GDPR, all those types of, things you can do your own rejects files and find that important information, that information that should be in a secure place. and then allowing you to hit your audit, your audits and things of that nature. That makes sense.
Bill Russell: Essentially, you're applying AI it to the data as it flows through. Is that accurate or am I overstating it?
Zak Pellecchia: It's [00:22:00] classification, so it's really built off of regexes, but we're giving you. The templates that are the most common within HIPAA requirements, let's say so rather than build your own kind of HIPAA template, we've got some prebuilt ones for you and you can add your own after the fact, we do use machine learning for the ransomware detection. so we're looking down to the directory level, to the file level and saying, Hey, this is unusual or anomalous behavior. Let's flag this and have you look into it.
Bill Russell: Thanks for not telling me I was wrong, but [00:22:30] I, what I heard was I was wrong, but I appreciate that. it's fine. I'm wrong on the show often and I, sometimes I refer to the show as the education of Bill Russell. I appreciate you guys educating me further on this stuff. Michael, in our earlier conversation, I found it interesting. That, we were talking about, running Epic in Azure and you guys just did a pilot and I want you to just share it easily become a full podcast in of itself.
I want you to just share it to just wet people's appetite [00:23:00] as to what the future might hold for running the EHR in the cloud. Could, can you share a couple of details around that?
Michael Friesen: Sure. so we have switched from our DR location for Epic being our secondary data center to being Azure. So in order for us to really say that is our DR
location, Epic kind of expects that you actually test it out and use it. [00:23:30] So as part of that, just a, about two weeks ago, we ran for our Epic prod environment out of Azure for a week. We wanted to gain some insight on a couple of things there. And I should say that this was a week of running out of Azure, but it was almost a year.
it was a year and a half of work to try and make that a reality. So it was not a simple undertaking and it [00:24:00] took a lot of help from Epic and from Microsoft to make that work. but. We were able to run for a week out of there. We wanted to try and figure out, you can look at all of the costs, calculations and so forth.
All you want to try and project what it's really going to cost. But the reality is there are so many. Extra things you don't always plug into those calculators. We wanted to see what the cost really would be. one of the challenges [00:24:30] around running Epic, I think most people will agree is chasing the latest hardware. that makes the cloud very attractive to us. If I can just have a quick skew change and now I'm running on the latest hardware, It's a lot cheaper. It seems like it's a lot cheaper than having to replace hardware. One of the things that came out of last week's test was a at the moment that [00:25:00] our Citrix, performance was actually better than on-prem.
And again, that goes back to. Our hardware is getting a little older it's time to replace it, which really breaks my heart because it's only three years old. But. that's the pace that Epic drives hardware at. So you can't say we're going to buy this and it'll be good for five years, any longer, you might get two years out of it.
And that's seems [00:25:30] unsustainable.
Bill Russell: Ha ha, I'll actually close this part of the conversation with this question, which was, did your users give you, What was your user's response? did they. I obviously they knew you were going to pilot this, you would have told them, but was the performance good enough that they weren't like calling every day saying, Hey, when are you switching back? Or how did it go?
Michael Friesen: So from what I have heard and they try and not let me out to talk to people too much. nobody [00:26:00] really noticed, so that is way better than I expected. Yeah, that would be a win. If you could fail over. And, and the performance is as good or better than what you have on prem.
Bill Russell: That is, that's a huge win. generally. I will, leave you guys the last word. Is there anything I didn't ask or didn't cover or some topic around, data protection and data control that I may have missed [00:26:30] or some aspect of the topic. I will take it from your silence on this that I asked every question.
Josh Peacock: I think you did a good job covering it. And Michael, again, congratulations on your guys's success in Azure for helping pave the way for some other clients that we're working with right now, it's much easier because of what you guys did.
Michael Friesen: Yeah. Yeah. The, the first person who runs through the wall, we all appreciate it takes you a year and a half to [00:27:00] do it. There's a lot of bumps and bruises and those kinds of things. But when we get to the wall, we just luck and somebody goes, Oh yeah, don't run through there. Just go through the little opening that they made for you over there.
So yeah, we. We do appreciate the early adopters and getting through there. I think there's a whole episode on this. I'll let, I'll let it play out a little bit and you guys, get some more learnings around it. And I definitely would like to explore the, EHR on Azure and, In AWS and other things and how [00:27:30] organizations are approaching it.
But this was, this was really helpful for me. I understand ransomware a little bit more. I understand how the attack happened. I understand how to protect against it. the challenge of, growing operational data stores, as well as the. the blob storage and other things that are going on it's, it continues to be one of those things that is, ever evolving.
You would think something like this, we got rid of tapes, which is great, but now we're, we're getting ready for whatever the next set of [00:28:00] challenges that we are going to face. Gentlemen, thanks for your time. I really appreciate it. Thank you. That's all for this week. Don't forget to sign up for clip notes.
Send an email to clip email@example.com special. Thanks to our sponsors. Our channel sponsors, VMware, StarBridge Advisors, Galen Healthcare, Health Lyrics, Sirius Healthcare, Pro Talent Advisors and HealthNXT for choosing to invest in developing the next generation of health leaders. This show is a production of This Week in Health IT. For more great content check out our website thisweekhealth.com. Or [00:28:30] the YouTube channel, if you want to support the show best way to do that and share it with peer. Actually another way to do that, sign up for clip notes and forward clip notes onto him and say, this is a great show. I'm getting a lot out of it. Please check back every Tuesday, Wednesday, and Friday for more great content. Thanks for listening. That's all for now.