This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

 Welcome to this Week in Health It where we discuss the news information, emerging thought with leaders from across the healthcare industry. This is episode number 29. Today we do a deep dive. I. This, uh, this one is in the cloud infrastructure and migration with our sponsor health lyrics. Find out how to leverage the cloud to new levels of efficiency and productivity.

Visit health lyrics.com to schedule your free consultation. My name is Bill Russell, recovering Healthcare, c i o, writer and advisor with the previously mentioned health lyrics. Uh, before I get to our guest and update our listener drive, uh, I recorded a brief segment yesterday with Shauna Smith, the executive Director for Hope Builders.

And, uh, I'm gonna play that right now for you. As we're coming to the close of our campaign in support of Hope Builders, an organization that gives disadvantaged youth the job training and life skills that they need in order to succeed. I wanted to share a little bit more about the organization with everyone.

Today I'm joined by Shauna Smith, the Executive Director of Hope Builders. Good morning, Shauna, and thanks for joining me today. Hi, bill. It's really a pleasure to be here. Thank you so much. Well, I'm, I'm really excited to have you here. Our, our great community, as I shared with you of listeners, has responded and raised close to $2,000 for Hope Builders over the last couple of weeks.

We still have a couple more weeks in our, uh, campaign. Our sponsor has agreed to give, uh, a thousand dollars for every a hundred new podcast listeners or, uh, YouTube subscribers. And I just thought this is a great opportunity to connect with you and, and give the audience a little bit more idea of what you guys do.

So I, I'm just gonna throw it to you with, uh, tell us a little bit about Hope Builders. Where'd you come from? Who do you serve and, um, what, what are you doing in the community that we live in? I think I'll start with the, where we came from, question. Um, 'cause it's a really, I think, compelling one and a unique one actually.

Um, you know, almost 25 years ago, a group of four Catholic nuns, the Sisters of St. Joseph of Orange, were living in a really tough neighborhood in. City of Santa Ana in Orange County, California. Not always a place thought of as a place of high need, but there are definitely dense pockets of that, and that's especially true in the early nineties.

Violence and youth violence were really, um, prevalent at that time, and the sisters were really caught in the middle of that and realized that for many young people there weren't a lot of options outside of the neighborhood. And so they were compelled into action. And as is our tradition, they engaged the community.

And 23 years later, we're serving, uh, close to 600 young people a year providing workforce training, really seeing jobs as a great vehicle for helping young people who are, um, really disenfranchised, um, move out of poverty and build lives of self-sufficiency for themselves. So that's really what we focus on.

You find these, these, this, these youth that are struggling and you give 'em a second chance. So how do you identify them and, and how do you, um, you know, and what does your program do for 'em? You know, so we really try to seek out those young people who aren't in school. They aren't in work. We're, we're particularly focused on young adults.

So that 18 to 25, 26 year old who's struggling, Um, and stuck maybe not in work or in a minimum wage job going nowhere, you know, and, and where we are in Southern California, like many parts of the country, it's very high cost of living. So we wanna ensure these young people get into the workforce and we're particularly focused on foreign industries, construction, healthcare, IT, and business.

We provide training and then we partner with local employers to provide them employment opportunities, um, which really kind of help enrich and grow their businesses. So it's a win-win for both the youth and our local employer community. Well, ai, I want to thank you again, uh, for the opportunity. I wanna thank you for the work that you're doing, and then it's such a great opportunity for us, uh, to support.

And I, I really appreciate our listeners who have subscribed to the podcast and our sponsor for, uh, for, uh, giving that money. We have another four weeks of the, um, Of the, uh, uh, fundraiser going. So I would, you know, just encourage people to share it with their friends. And, uh, let's see if we can, uh, drive the number higher.

And thanks again for, for coming on, Sean. I really appreciate it. Well, thanks to you and your listeners for your generosity and the opportunity to share about our good work. We really appreciate it. I appreciate Shauna coming and, uh, joining me and, uh, and I also really appreciate our listeners that have, uh, spread the word about the show and we really appreciate it.

As I said, we're, uh, we've raised $2,000. We hope to raise more in the next couple of weeks, and, uh, we appreciate everything you guys do. Um, with regard to that. So, uh, today's guest is, uh, Robert Rice, who is the Chief Everything, chief architect, chief Technology, chief Guru, chief Operating Officer, uh, for health lyrics.

Rob and I have worked together for a while. Uh, good morning, Robin. Uh, welcome to the show. Good morning. Thank you. So you have a, you have an interesting background. Um, you've worked in a lot of, a lot of different industries besides healthcare, uh, what are some of the other industries you've, uh, you've worked in?

Uh, so I started in finance and then prior to healthcare, I, I worked for the entertainment industry. So the Walt Disney Company, for example. Walt Disney Company. So you work for Disney. What's, what's it like to work for Disney? Is it, is it as magical as we would think? Uh, it's a lot more intense than magical.

Yeah. It's, it's intense for the people that make it magical for the people who are visiting, I guess. Uh, yeah. What's, what's the, what's the craziest it project you had ever had to pull off at, at Disney? Just outta curiosity. You know, it is not that there's one intensity over the other, it's size. Right. So, um, so for example, an identity project at a normal, large institute might have a hundred thousand identity accounts.

At Disney, it was about 2 million identity accounts. So just sheer volume alone caused you to rethink and re-look at the way you were doing things. Wow. Was, was there anything about the, in terms of down would. Disney had like a zero tolerance for downtime kind of situation. Yeah. And it was based in the culture, right?

Everybody knew that we worked when most people were off, so we had to make sure that we didn't impact that. And uh, it was kind of mantra that we only have one time to get right for the user experience. For the family experience, we kind of measured th rice cut once. So that was, uh, that was good. Good training for you to, uh, to, to come into healthcare.

So then you came into healthcare, you actually, uh, joined me at, at, uh, St. Joe's in Orange County. And um, and that's really gonna be the gist. We're gonna talk a lot about, uh, that time there. And then when the Providence merger happened, you and I went off. And, uh, full disclosure, we are, uh, partners in, uh, health lyrics and, uh, we've been doing cloud migrations now really since about 2010.

Uh, not only with, uh, St. Joe's, but with other health systems. And that's the, the, just this, so this is a deep dive episode, which means, uh, we take one topic and really explore it. We've, we've done this before with ai. And, uh, now we're gonna do this around, uh, cloud computing, but more specifically, we're gonna focus in on cloud infrastructure.

We could talk about cloud data models and some other things, but this one's gonna be about cloud infrastructure. Uh, I'll do the, the setup and then we can move deeper into the process. Um, as you know, as I've said, I, I, you know, Rob and I have a lot of the same shared experience with this. Uh, plus some of this, my talking more on this episode is just gonna be.

Just sympathy for Rob 'cause his plane got into L A X after midnight last night. So, um, we really appreciate , I really appreciate you coming on the show. Um, maybe not at, at the top of your game right now, I guess so , so let's, uh, let's start off with a quote. This is pretty recent. This is happening this morning.

So, uh, Dale Sanders, president of Technology with Health Catalyst. Uh, had this to, had this to say on LinkedIn this morning, uh, roughly 35 minutes ago. If and when I ever returned to the c I role in healthcare, I'll raced as fast as possible to move my data centers to the public cloud. There's no way I can provide the same capabilities on any dimension with my data center.

Even the dentist data centers like ViaWest can't come close to Mac matching Azure, a w s, et cetera. Um, and I think it's interesting, I think you're seeing a lot of, uh, John Halamka had, uh, on the show also had similar, uh, comments about, uh, the, the capabilities and things that are available in the cloud.

So, um, So, let me set this up. So let me put a little definition around it. Uh, I came in as c i o in 2010. I, I looked at the organization, looked at Rob and said, you know, we're gonna, here are the characteristics of cloud that we're looking for. And this was back in 2010. Not a perfect definition, but. It, it sets the groundwork.

Uh, it, we said it has five characteristics. First, uh, self-service provisioning from anywhere. So, uh, web-based provisioning really from, uh, from your phone, from Alexa, from, you know, self-service, provisioning from anywhere. Uh, that was number one. Number two, rapid elasticity, which meant you could scale up and scale down.

Now, scaling up was not that big of a deal back then, but, uh, but it was somewhat, sometimes you had to add. Uh, you know, crack units and air conditioning and power to your data center, in which case it was millions of dollars. Uh, other times you had to add servers. Took a lot of time. Uh, but one of the things that we were looking at, uh, when I came in in 2010 was the ability to scale down, uh, the, or.

We kept asking the organization for tens of millions of dollars for new data centers and whatnot, and they were saying, okay, hey, we're . We're actually selling off portions of our business. Can we have some of that money back? And the answer is no. So cloud gives you the ability for self-service provisioning and rapid elasticity.

So scale up, scale down, uh, meter service, pay for what you use. So, um, one of the benefits of the cloud is if you have a a thousand users today, you pay for a thousand users. If you have 800, you're paying for 800. But even more than that, The, uh, the services are much more granular, so, uh, you're only paying for the storage you need and the compute you need.

And now it goes all the way down to, uh, you know much, you know, AI and other things as well. Uh, the fourth thing was programmatic access to resources. This is where we get into the DevOps. This is where a lot of your efficiency comes in. And then the fifth thing was granular accounting, um, to the point wheres now.

You can do compute by the minute. So, uh, for those of you who are wondering what you would use that for, sometimes you have, uh, processing that's done at night. You can scale up a hundred servers, use 'em for 10 minutes, do all your processing, scale 'em back down, and only pay for using those servers for 10 minutes.

So, so granular accounting, so cell servers, provisioning, rapid elasticity, metered service, programmatic access, uh, to resources and granular accounting. And, uh, like I said, that that, not a perfect definition, but it sort of set up what we were looking to accomplish. We were looking to get agility, efficiency, and automation within our, uh, health system.

So we thought the cloud was gonna bring us really, uh, two important things. First in 2010 when I came in, we, we interviewed a lot of the executives and said, where is healthcare going? And the most common . Uh, response was, we don't really know. We know that there's a lot of change coming. Pop health, value-based care, uh, mergers and acquisitions, new competitions, Silicon Valley was coming in.

Um, and uh, you know, one of the things is when you don't know what's next in your industry, you need to create something that is. That you're able to make quick moves, you're able to adapt very quickly. And so cloud gives you that ability. Uh, the second thing we knew is that healthcare was moving outside the walls of the traditional health system.

So we used to have data centers, which served our 23,000 staff. Uh, but that wasn't going to be the case moving forward. We were gonna start putting devices all over the place. We also had clinics, uh, critical. Facilities more and new types of partnerships so that, that not only were we looking at 23,000 endpoints, we were now looking at literally millions of endpoints, uh, accessing resources from our data center.

Uh, and the cloud lend itself pretty well for that. So, uh, one last piece of background, Rob, before we get going. So, um, just to give people a little idea of where we started our journey. Um, you know, our, our scenario was pretty challenging. We had, uh, here's some of the, the statistics. So 50% of our data center equipment was end of life.

Uh, when we started our project, 60% of the workstations were also end of life, uh, 1800 instances of 900 distinct applications, which is not the worst we've seen in the industry. I know you, you and I have seen, uh, organizations, when we say 1800 instances, they just go, oh, well that's, that's nothing. We have 10,000.

And you just sort of shake your head like, That's, uh, almost impossible to manage 10,000, uh, of applications, 18 applications. Uh, we had 15 million in data center upgrades needed to. And, um, the data center, I mean, this is the, the real thing here. The data center had just gone down eight times in six weeks prior to, uh, to my arrival as c I O.

Uh, it's actually one of the things that precipitated my, uh, coming in as c i O. So, uh, and the other thing I will give people just 'cause it was, it gives a picture of our relationship. I was the unreasonable c i o you were the VP of, uh, infrastructure and operations. And I looked at you and said, Hey, you know, get our data centers to the cloud.

And oh, by the way, here's one last thing. You can only have two racks per hospital and they have to be self-contained racks and not need data centers. And I remember, uh, actually I told the team that before you, before I hired you, and they looked at me and said, that's not possible. And then when I brought you in, I said, Hey, here's the requirement.

I thought you were gonna tell me the same thing. And you looked at me and said, oh yeah, piece of cake. We can do that. So we'll, uh, we'll shed some light on why that was a piece of cake and actually how you exceeded, uh, that expectation, which was pretty interesting. Alright, are you ready? Yes. Great. So we, we've set it up.

It's a, obviously it's a picture perfect environment. Most of the stuff's data centers going down. You come in as VP of infrastructure and operations and, uh, we cloud strategy. You take the cloud strategy, but one of the things that we talked a lot about is we're not moving to the cloud for cloud's sake.

We're looking to really, uh, derive benefits for a healthcare organization. Um, let's start out there. So what are some of the benefits that a healthcare organization will get from moving to the cloud? Um, oh, there's so many. Where to start, right? And first and foremost, it gets you out of the client. Like IQ client.

You're a healthcare business. You're not an IT business. You shouldn't be there. We don't do it well because we, we don't attract the level of talent that we need because honestly Right. Most of that level of talent wants a, Google wants a Disney wants an Amazon on their resume. Yeah. And in fairness, in fairness, we were in Southern California, so we were competing with, literally with Google, with startups, with Disney and others.

So it was, it was a tough labor market. Right, right.

Is also not very complex in the grand scheme of things. So it lends itself easily to get yourself out once you get over that one kind of emotional hurdle is, you know, I need to own the servers. Yeah. And we did have people that that wondered, you know, if we move to the cloud, how are we gonna walk into the data center?

I mean, how, how did you address some of those, some of those concerns and, and I mean the benefits far outweigh that, you know, being able to walk into the data center. Yeah. Well, first and foremost, right? It's, it's understanding and proving, you know, do we know what we need to know and what do we don't know?

Right? And kind of funny speak, but the, the basic ism for it is what is our footprint? What are we actually management, and what do we think we know versus all that that doesn't around. It, it becomes an easier picture to paint that we probably don't manage enough of it already, so it's easier to start kind of boxing up and shipping out.

Yeah. Between you and, and David Baker. David Baker, who we had on the show earlier, uh, between you and David, you guys kept reminding me that, um, you know, it's, it's not compute storage and, uh, and those things that people buy, they buy, uh, what they're, what they want from it is uptime, uh, ease of use, access.

Um, they're buying services. And so much of our, uh, challenge upfront was to get it to think different, get, get the culture to think different of what's the best way to deliver service to the organization. Not what's the best way to get, you know, storage, compute, and all that stuff to the, to the business.

Right. Yeah, it was, it was interesting cultural change. So we, we, we saw, uh, financial benefits, um, You know, our, we, once you get to DevOps and everything's programmatic, we went from 35 down to five. Uptime went up. The, uh, the, uh, overall costs were, uh, more tied to, uh, the level of the business. Uh, and then I think the other thing that people are surprised to hear is that, Um, our, our availability actually goes up in the cloud, and we're gonna talk about that a little bit, uh, later.

But, uh, you know, you have disaster recovery, uh, capabilities that aren't available in traditional models, uh, that you were able to leverage on a regional basis and on a, uh, on a system-wide basis, which we're, uh, We're pretty interesting. So, um, alright, so let's, let's just, we'll step into the process here.

So, uh, and again, this is the process that you gave me way back when, um, There's really three movements to get us, get us to the cloud. One is to establish a business case second, uh, to architect and plan, and then the third to migrate and operate. And in under each one of those, there's a, there's a handful of, uh, steps and movements and, uh, we're gonna touch on those.

So the, the first thing under establish a business case, you have a readiness assessment. And, uh, I think the thing that people will be interested to hear is not every health system is ready to go. So, uh, why is that, why do you think what makes one health system more ready than another to, to, to move to the cloud?

Uh, I don't think we've already talked about this, right? And at least the, the simplicity of it, right? I think the context is organic growth versus, you know, measured growth on that. Most health systems that I've seen have grown organically, like, I'm gonna buy this system or that system, or that system. And just keep adding on to the bandaid ball, but as opposed to, you know, logical and methodical growth.

You know, we know we're gonna grow by a factor of X year over year and we're gonna plan for it. And I think that's kind of really at the epicenter of Y from what I've seen. Because coming from other industries we've done that, you know, it's coming into healthcare. I'd yet to see that logically thought through.

Yeah, I, I think the other thing we, we, we've noticed is that some health systems still don't have, uh, a handle on the, the basics, on the, on the basics of blocking tackling. So when we go in there, like for example, um, I. You know, when I came in as C I O, I asked, okay, I would like a list of all the organizations that we're connected to and what data we're sharing.

Um mm-hmm. , that took almost six months to a year for me to get that list. So organizations that have trouble with the basics of, Hey, can we have access to this data, this information? Do you know where your contracts are? Um, Those organizations need to, I mean, not that it's an insurmountable hurdle 'cause we got, we got through it and, um, but, you know, being able to have access to that data, even though it's, you know, a lot of times we can just take raw data and, and process it.

Um, just knowing where it is and, and having access is, is one of the things. So, uh, you know, what are some other things you, you found in terms of organizations being ready? Uh, I think process maturity is a key, right? So standardization, and this is not process for process sake, is do you have standard repeatable processes that a large portion of your staff follow?

And in a lot of cases it doesn't exist, and that's the baseline for building blocks of automation. And kind of growth from that. So, you know, building on what you said, like asset management is one key, but do I have standardized procedures for patch and reboot for change or things like that? So, so if it's measured and applied consistently across a larger part of the organization, you're much more mature and ready to take those leads rather than you have to set those building blocks into place first.

So if they've done ITIL and those kind of things, you're, you're, you're gonna be in good shape, I would assume, right? All right, so the next step is to establish the point of truth. And, uh, you know, I was, I found this interesting because, uh, you know, for me, you, you came in and started dumping all this, uh, information on me and it didn't match with what the organization had told me.

Um, you know, prior to, to you doing this deep dive into our, uh, information. And, uh, and to be honest with you, I was, I was a little defensive because I, I'm looking at. We had just done a Microsoft agreement based on information that Microsoft had given us. So they had come in and done an audit and said, Hey, here's all the stuff you have.

Here's what you should license. So that's what we license. And then you came in, did this deep dive and came in and said, Hey, you over licensed by about, you know, a million and a half dollars. And I was, and I got defensive because I'm like, how can you say, You know, more than Microsoft who has all these tools on my environment, but give us an idea of what it means to establish the point of truth and how you get to, um, really that, uh, that starting point for the cloud migration.

So a lot of your, we'll say your source logical systems have the data of how you actually use. From a licensing perspective or consumption perspective. So active directory for example, in there you have every user that's gonna log in and every account that's gonna do it, but not every account is active.

So when you start looking at it like how fast and frequently do I actually decom or kind of terminate my accounts in there? And that, that was one of the problems that we had is, you know, we had approximately 55. 15,000 of them have been fallow for multiple years, meaning that we don't use them at all.

So we're licensing for the wrong amount for that. The same thing applies, you know, in most other areas, right? So we never do a good job of rationalizing down, right? We, we can buy, we could acquire, we could build, you know, but what's our lifecycle management process? And that's kind of one of those, you know, easy sacred cows to go after, first and foremost, to say how much have we been decommissioning over the years, which.

I think I found it interesting that you, you go after source data. I mean, you, uh, literally are just saying, Hey, just, just dump me the raw. I, I just want the raw data out of active directory. I want, uh, I wanna look at the physical contracts. You, you, you really, this process, you go back and one of the things you, you sort of drove home to me is that every report I'm seeing has a bias or a purpose.

A purpose or a bias. It has a bias to make something look a certain way or it has a purpose that it was designed for, and I might be using that report for a different purpose. Um, so talk about why you go back to, to source data and, and how, uh, and why that doesn't take forever. I would think if you go to source data, it would just take forever to, to get that done.

Well, I mean, and you're right.

Performance. Asset reports are just that they're generic. They don't apply to the way you actually use it. They apply to the way the tool is done. So when you look at the problem from a tool perspective, you're solving a problem with a hammer when you actually need to build a house. When you go to the raw of the data, you're looking at actually how you're using it and which is really the key, which takes a little bit more time up front, but it moves a lot faster once you lay out all the cards.

So like when you're solving, uh, putting together puzzle, for example, you typically look at the edges rather than starting in the center. You. And that's kind of the approach where you have to at it, how am I actually using it, and what am I actually doing with first? Which us move faster once you get the edge of the puzzle.

Yep, absolutely. And so, you know, to, you know, the, and then we, you know, you have this point of truth and then you establish the. The, the business case itself, uh, which closes out this first step. And in the business case, you have things like financial models, you have usage, uh, you have skills, you have, um, uh, you know, I'm, I'm just, I'm, I'm actually shortening this up for, for purposes of the show, but, uh, there's a lot of elements of this business case.

Uh, one of the things that, the combination of you and I really works, as people can tell, you're very analytical and I like to tell stories. And so, I really strongly encourage people to tell the, the whole story of how healthcare transitions in this process. And then you come back with just all this supporting data, uh, in terms of, uh, how, how agility is going to play out in the organization.

How, uh, there are cost savings. Uh, even though people need to be aware of the fact it's, it's operational dollars versus capital dollars in a lot of cases. And some organizations struggle with that, uh, transition. So we, we now move into architect and plan, which is the, the second step that you have here.

And the first part of that I think is probably one of the more fascinating, which is decompose the applications and the services to core elements. Uh, what are you actually doing there? Uh, you, you're just looking for the commonality across your application portfolio, right? Every application has the same components.

You know where the data exists, how does it process, what's the client side look like? So all we're doing is we're taking that and establishing a commonality across the largest portion of the application portfolio. We could start standardizing and really focusing out, because you're not gonna solve problem individually.

If you have hundreds, thousands of applications, you're gonna look for the least common denominator across the group, which allows you to do that. So, uh, database, for example, most applications have a database. In an on-premise legacy world, you have your production database plus your backup, maybe a, a high availability one.

So now, but from a core conceptual, it's all allows us to start reducing down. And solving the database problem rather than, you know, the thousand application problem. Yep, exactly. And it's, it's interesting 'cause you, I think you had 13, I, I forget the number of layers, but it was, uh, you know, it's everything you would think about.

You have the fiscal hardware, the networking. Uh, the virtualization layer. I'm gonna get some of these wrong storage layer, uh, os and whatnot. And when you got done that, one of the, one of the big findings I found was you came over and said, Hey, we have, uh, it was a couple hundred images. I, I, I forget the number.

Do you remember the number of images, server images we had? It was a little over 200 server image bills. Yeah. Yeah, so it, so that's one of the things, when you look at it through a different lens, you're able to see this, wow, we have a couple hundred images. Well, that's very hard for an organization to maintain that level.

And then when you broke it down even further, you, I think what your team came back and said is immediately we can get this down into the teens because of, you know, if we just do some work around the images. And eventually, I think the goal was to get it down to six. Is that, is that pretty close? So about 200 down to five common images.

And based on that, now you could automate, you know, you could start leveraging efficiencies because now I'm not spending time chasing after 200 different, that 40,000 server types. Uh, and chasing down just five images, exact same. And it was really interesting. Before you came in, we were chasing this C M D B.

Uh, Nirvana where we were gonna have the c m db, which would always tell us what was going on in the servers. And, and you just, you sort of minimize that. You're just sort of like, Hey, you know what, if we get down to five images, then, you know, it's, it's, the CMDB becomes not irrelevant, but becomes, first of all, a lot easier to manage.

But second of all, I mean, it's one of five images. You don't have to figure out all the things that you would normally have to dive deep into A C M D B to figure out. So, um, mm-hmm. , it was, I, I found that to be interesting. So the next thing, and this really highlights a mistake that I made prior to you coming in, is the skills.

So I skipped this step, and one of the things people need to know is that the biggest obstacle they're going to get to moving to the cloud is not from the organization. The organization's gonna say, Hey, I use the cloud at home and I got 25 giga storage for my email, and I use Dropbox or Box at home. And I, you know, I have all these capabilities and they understand that cloud gives them better services.

So they're not, they're not pushing back to you when you say, Hey, we're moving to the cloud. It does. And one of the things that I skipped was the skills matrix part. And because I didn't, people didn't see where they fit in the organization when all this was done. And, uh, and, and so you, you made a step back.

You created, um, you know, a skills matrix for where we were at today and what skills they would need to develop to get to, uh, where they were going. Uh, so talk about the process of not only developing the skills matrix, but also having the conversations with the employees. Uh, and the, and the staff around, um, moving to the cloud.

Well, uh, first and foremost, right, the, the skillset shifts or elevates up in the cloud because again, you're not emphasizing the petty engineering skills. You don't need a lot of really core storage depth and knowledge. 'cause all you're doing is spinning up gigabytes and terabytes of data at a.

We never really defined what success was from a growth roadmap, from a skill perspective. So we deemphasized planning, design, you know, reporting those types of skills that you actually really need more so, 'cause you need to map out more intensively. So when we created the skills matrix, you know, we rated all the skills that we're gonna need in different types of roles on a one to five basis and soft skills.

So we kind of reset what success was for each individual team and group to say, you know, right now you're engineering heavy, but we need you get, we need it. Management mapping, things like that because that's where it really kind helped us. And not everybody wants to do that or takes that leap to do that as well, so.

And, and that was the fascinating thing to me. I mean, you sat down with a significant staff and started mapping that out, and some of them just self opted out. You looked at 'em and said, Hey, here's where it needs to get to. And they said, you know, I'm fine. I'm fine working with cobol. And you're like, I.

Okay, that's fine because we're gonna have this legacy environment for at least a year, maybe a year and a half while we do the migration and we need somebody to take care of it. So, you know, but at the end of a year and a half, we're not gonna need COBOL programmers. I'm using that as an example. COBOL programmers.

So you've essentially self opted out over the next 18 months, but you know it. But people appreciated knowing that of just, okay, so what you're saying is, my job's gonna go away in 18 months, but you need me for the next 18 months to get there. And they, they were really appreciative. Um, and I, I would just strongly encourage people do not , it seems so obvious to CIOs, but for some reason I skipped it.

Uh, you have to show people where they're gonna fit in the future state. Otherwise they, uh, they will fight, fight the process. So, uh, yeah, you also need to, those soft skills and those kind of intangible.

There's no step-by-step guide, so like how do you identify your innovative thinkers, your testers, your TERs, because part of the process is learning self-learning, and if you need a, you know, C B T or a training module to do that, you're probably not the right fit for that kind of role. Yeah. And so some of the most people we found came out of the most unlikely places.

Started doing that. Yeah. It always surprised me where you were, you, you were, were moving people around and I'm like, yesterday that person was doing that and now you have 'em doing that. Are you kidding me? And you're like, no, let me tell you how these things match up. This is, you know, this is the new world.

And. Um, I think the other thing is we just had a conversation with a, a client with a potential client, actually, I should be honest there. So with a potential client, and they were saying, are you saying in the DevOps world, I'm gonna have to hire all these programmers? 'cause that's gonna be really hard for me.

And I, I think you, you, why don't you, why don't you talk about that a little bit? 'cause I think that's a common misconception. Uh, yeah, I think it's, the inverse is true, right? It's like once you have DevOps, you're gonna plan to lose them, right? Because they're going to the bigger names, they're building, you know, their, their references and their resumes based on that.

So what you have to do is kind of create more of the, the grooming and training on, just like we talked about before, it's like how do you invest in the people that need the time to learn and train and, and develop their skills to do it? Because you can't compete with those big guns in the industry. So when you kind of give them a growth path to do that, you have the ability to, to really kind of build that skillset.

And then the other thing we're finding is that the tools are becoming more configuration based and less, you know, hardcore coding based. Right? So it's, it really is going into an interface and clicking on a couple things and then boom, you've just, You've just stood up a whole bunch. Not that you don't need a couple of those hardcore programmers who really understand how the underlying, uh, framework works, but a lot of the, uh, tools are, are, are moving higher up in the, in the, uh, uh, in really the, into the presentation layer.

So, uh, so people who just understand how that entire stack works can make that stuff work. Um, yeah, like when we started, we used to, and I'm talking like earlier, mid 2005 ish or so, you had to have core development skills in order to do this, but now like 50 to 75% of the work is configuration based.

There's still an element of development in DevOps, you know, but now it's kind of funneling higher and higher. Absolutely. So the next thing, and uh, you know, probably the sexier thing here is establish a digital architecture security models and align workloads with the appropriate, uh, deployment models.

Um, and what, what we talk about a lot here is that each cloud will look a little bit different from client to client based on what they're trying to accomplish. But you do have five primary environments. Today. Today. Um, can you, can you so walk us through the five environments again. I mean, you gave us, so there's a, there's a legacy environment, uh, and I mean, you know what list I'm talking about?

Yeah, yeah. Okay. Yeah. So, um, so the legacy environment is just, That not all your systems are gonna out on that, so you have to maintain environment. So why is that? Is that just, is, is that just because because of the, the way they're written and, and the way they, they function today, they, they're just not gonna go to a cloud environment.

Uh, yes. Right. So there are some devices that still require direct physical connection. So in the health industry, biomed devices, there's, you're just not gonna get outta them anytime soon. Then there, the larger ones that are critical systems like your m r, that your vendor won't support different processing models, right?

Because they're monolithic in nature. So you may as well just acknowledge them, put in the, come in and focus on. Okay. As you, you're scattered. So the first one we have is a legacy environment. Uh, walk us through the rest. So the next level up is you have applications that are still server based, but they're virtualized.

So a virtualized server can exist anywhere. It doesn't have to exist in your data center. Yep. For the most part. So, and, and those are the next target to move out into infrastructure as a service. So your Azure bmm hosting your AWS instances, but they're still server based. So you're building instances that, and then when you take the next step beyond that, now you're focusing higher up, which is more workflow.

So you're going after software as a service platforms, which everybody has right now. So whether you're using, um, Salesforce or Dynamics or you know, other software based platform, you know, you're focused on workflow there. You're not focused on anything below the workflow for the most part. And that also kind of lends itself to platform as a service too, right?

So when you're making those investments into SharePoint online, into Salesforce, into, you know, other C R M platforms, now you have more platforms that I could kind of build different workflows, different processes, different solutions in a single platform instance rather than different software instances.

And then ultimately the vision when people get to that level is native development, cloud native development models where you're designing microservices, web services and the like. But that's typically the the end state of a journey. So the, so the distinction when we go, so every, every health system will have those five components, legacy or a private cloud, infrastructure as a service, software as a service platform, and then cloud native.

So what's the distinctive is how much of each of those you're going to use. So some, some health systems might have 20% legacy, uh, you know, 30% infrastructure as a service and so forth and so on. Um, but the goal really is whatever, whatever that that core layer of legacy we're, we're shrinking that over the next couple of years.

We're moving to infrastructure as a service. Even that layer, we're moving to software platform and cloud native. Um, and the reason we're doing that is the, the levels of efficiency, agility, cost savings, and uh, access to innovation that organizations are gonna have by going to the cloud are just, uh, significantly more.

Um, than, uh, traditional, traditional models. Alright, so we create these really cool diagrams. We map out, uh, the entire, uh, digital architecture, the security model, and, uh, we align the workloads to the various, uh, layers of, of cloud infrastructure that we're gonna use. Now we're ready to migrate and operate.

And, uh, this is the easy part, right? So we, uh, we, we just have to migrate 1800 application, 1800 instances of applications. It's pretty straightforward. Um, and, uh, I'm being a little facetious here. So here's where we, here's where you start this process. You start with, uh, negotiating agreements and KPIs with the, uh, cloud vendors.

What are some of the things that a healthcare c i o, or healthcare organization should be thinking about when they're negotiating these cloud-based contracts? Um, I, I guess first and foremost, earlier conversation on like, knowing your environment, knowing.

Virtual servers or instances I'm going to need to maintain because I just can't get rid of them. But they're still server based. You could lock in certain pricing models with your vendors, right? So, so when you're locking in reserve pricing for, I'll use, um, Azure for example, and say, I know I'm gonna maintain 5,000 VMs.

You're gonna get a very different price point. Then if you say, you know, I'm gonna spin up instances willy-nilly, you know, and it's a vast order of magnitude too. So knowing what your reserve pricing and forecast will be, you'll say, I know I'm gonna maintain this for least three years. Allows you to negotiate very differently than just kind of winging it and then going into it.

So again, back to that very first step, which is knowing your world and what you're going, and then from there,

Yeah. Well, and some of the things that, that we talk about a lot is you, you can't outsource security 'cause you have a background in security. So you can't outsource security. So you need to negotiate that in. Uh, so understand the HIPAA requirements, understand any P C I requirements you have, uh, and you have to be able to, uh, provide the reports and, and pass through all the way, all the way down.

So you can't outsource it. You're still responsible. You still have to.

The other thing, the other gotcha that I think people don't think about is, um, especially with cloud applications, like, uh, where, you know, you're moving vast amounts of data into these cloud environments. Um, make sure you can get it out. Um, you know, when you're negotiating these contracts, I. Uh, not only make sure you can get it out technically, make sure you can get it out financially.

Uh, I know that some of these clouds, it's interesting, they will, they'll literally send trucks out to pick up your data, bring it to their data center and pump it in. And some of those, some of the services almost free, but to get it out, they hit you with bandwidth costs and all sorts of other stuff, and it costs a, a fortune.

To get it back out. So be be careful of those kinds of things. And, you know, this is a, we only have a couple more minutes, but those are, those are just some of the things off the top of my head that I would, uh, I, I, I would say just keep those things, keep those things in mind. So this is the part, remember, to make sure emphasize too, is the type of data, right?

So,

Once in a long period. So your PACS image research imaging, for example. Yeah. Which allows you motivation different. Yeah. And when we're looking at, uh, when we're looking at the architecture, we're looking at those bandwidth requirements and we're looking at those access requirements because the, the, the more access you need in the cloud, you can pay for very high availability, very high access, very low, uh, bandwidth.

Or latency, low latency, sorry, low latency, uh, servers. But if you don't have to, you really shouldn't. So, uh, so really knowing your workloads and aligning, uh, correctly to the, to what you're buying is, is really important. So here's where I wanna spend a bunch of time, because this is, you spend so much time here and this is where the big payoff is.

So, um, you've, you've done all this, you've done this deep dive, you've done all this planning, and now we're getting ready to migrate workloads and, uh, You know, at, at first migrating the workloads goes a little slow, but then it really picks up pace 'cause you're just doing the same thing over and over again.

So give us an idea of some of the things you think about going into, uh, migrating to the cloud and some of the processes that you, uh, you take a look at. Well, um, first and foremost, if we're doing lift and shift migrations, the easiest way to do it that we always found is to do a disaster recovery failover.

So we create the D disaster instance in the cloud, which allows us to create a full production snapshot. Everything we need to build the skill, build the confidence, build the expertise for it, and then go through a few iterations of failover without impacting the business. So in those cycles now we've built everything that we need to at a low price point, at a low impact to the production environment, and got the business to buy in and sign off on it and build our, our total competence.

That's kind the probably the primary way that we start the process because it allows to confidence.

You really encourage people to start with the easy stuff first because your team is learning a new skill. So you, you, you, you don't want to start with, Hey, let's start with pacs . You really, you wanna start with something that's, uh, pretty, pretty small, pretty self-contained, um, not a complex architecture, and your, your team's just getting that.

Building up those skills. And then as you, as you sort of move down the, the road, then you start doing some of the more challenging, more difficult, uh, migrations. But I, I wanna go back to something you said. So you start by standing up a, a DR environment or a greenfield environment that you can, uh, fail over to, um, That, that really is, uh, one of the, one of the secrets that you brought to the, to the process that kept our, uh, kept our downtime pretty minimal in this migration process.

I mean, because when you can fail over from one to the other and, and just sort of switch those users over, now you're not looking at extended downtimes. Um, and plus in the process you're building out DR capabilities for each of these applications as you move forward. Um, So does that work for all the applications or just a certain set of those applications?

In a large portion of them, it does. Right. And, and I would qualify that to say a large portion of them is transparent, right? The user experience is barely a blip on the radar for them. There are some that struggle with that, but architecturally, you know, you identify them before. Yeah. And so I, I think the other thing which you drove home, you and and David both drove home to me was, uh, this migration process is more of a people and culture, uh, challenge than a technical challenge.

Um, obviously we're, we're bringing our IT staff along. We're training them as we go. I. Uh, but also you guys built up a pretty solid team, uh, with Rachel and, and the rest of the team and Rachel, who we've hired at Health Lyrics now, um, who, who put together great processes for engaging, uh, the end user community.

So they, uh, engaged them early in the process, uh, walked them through it, talked about the benefits, um, so that team. It became really good after, you know, after you do a couple hundred applications, they, they actually ended up with an extensive database of everyone in the organization. They knew who the key decision makers were.

They were able to engage them at every point within that, and they became a trusted source for that. So, uh, is that, I mean, do, do you still see that as a pretty critical team to have in place for this migration? Oh yeah, absolutely. It's not worth doing if you don't understand your business and if you're not talking with them, right?

Because they're gonna define their tolerances, they're gonna define their appetites. They're gonna tell you, don't bother me these days, but I'm okay these days. Which gives you your operating models to build your confidence for them. And there are so many other problems that you find that exist outside normal.

It that you win a lot of goodwill just by solving easier problems too. So building that relationship back, absolutely critical. You know, some of the, um, not necessarily seen benefits for us were, uh, I think we, we got rid of 10% of the applications just by talking to people. Um, just by, just by going out there and saying, Hey, we're gonna be migrating this application.

You're listed as the owner, and they looked at you like, uh, you, we don't use that application anymore. Great . We, um, and so we, we didn't know we were doing a, uh, an application rationalization, pro project, an appra project, but we ended up, uh, you know, archiving a whole bunch of data and a whole bunch of applications just in the process because as you pointed out earlier, a lot of healthcare just grew over time of we added this application, added this application, we acquired this company.

We never really cleaned all this stuff up, and this is a great opportunity. To do that. So we get 24 months later, here are some of the things, you know, we, we, we never did spend that $15 million on data center, uh, improvements because we didn't need to. Um, we went from, uh, server provisioning from, and people find this hard to believe, but really from.

Two to three weeks to provision a new server with the processes that we had to almost two to five minutes, uh, to provision a new server. And, and sometimes our clients find that hard to believe, but in a, uh, to be honest with you, with a credit card right now, I could go online and, and provision a hundred servers in the next, uh, 10 minutes on a w Ss, and I don't, you know, I don't even have the requisite stuff.

Lined up yet. So, um, that's the kind of stuff that's available in the, in the cloud. Um, and then the other thing is we, we got access to a whole bunch of new capabilities. Uh, you created a, uh, uh, the ability to fail over regionally, um, uh, and, and some other things. But I wanna get back to the, to one thing, because you, from private cloud, you went in a hyper-converged route.

Which I, I found interesting and I, I came to you and said, Hey, no more than two racks in every hospital. And I know when, when I said that to the team, they, they sort of were incredulous. And I think some, some of our listeners from health systems are saying, uh, you know, we, there's no way our health system, uh, in that hospital could get down to that, to that number.

Um, but talk to a little, talk a little bit about what Hyperconvergence was able to do, uh, at the hospital level, uh, for us. So all hyper compute is, is really condensed compute and storage memory power, right? So most applications that exist nowadays are nowhere near the level that you could provide, you know, in basically what I could fit under a desk right now.

So when we started getting into the applications and we. You know, these are my toys elements to the local staff too. So being able to come in and say, we're not gonna take your toys out of there, but we're gonna do it. Just put 'em in a new case. But that new case now is a fraction of the size, right? So this is kind of the, you know, just the convergence of data and compute.

So we found almost immediately that, you know, about six use, which is roughly, you know, six to 12 inches of space, could hold hundreds of servers of what you currently have. So getting to that point was extremely easy. It also kind of won us a lot of different confidences too by saying you're not moving it out.

And once it was in the hyperconverge, our ability to migrate out if we, for the DR stuff was almost. Yeah, absolutely. And then, and then you can go to third parties and be able to migrate out and migrate in as needed. Uh, and, uh, so yeah, you were able to get down to, in a lot of our hospitals, you're able to get down to a half rack.

I think the, the largest was maybe two racks. I, I know you always argued with me on that, but, um, but essentially self-contained racks. You didn't have to have a specialty room. You can move that stuff around. And I think that has some, uh, Dr. Um, DR. Benefits, uh, for it. Uh, the other thing is people, people always ask me, you know, if you're moving this stuff into the cloud, what happens in the case of disaster?

So if your connection to the cloud goes, what, uh,

What, you know, what do you do in that case? And so one of the more ingenious things I think that your, you and your team came up with was, uh, almost a reverse DR process, which is, uh, it was our primary was in the cloud, but we, we were actually, uh, trickling data back down into the local data center so that if, if the, if something went down, Um, on the network.

Let's say there was an earthquake in Southern California, the network went down. Those hospitals would still be able to run, uh, certain core systems because we had trickled the data back. So it was almost a reverse, uh, dr kind of thing. Um, hey Rob, I want to thank you for, you know, I know you got in late last night.

I wanna thank you for coming on the show. Um, and, uh, you know, it's, it's, it's always fun to, uh, to talk through this stuff with you and to, to be a part of. Uh, helping now other health systems, uh, to go in this direction, so thanks again. Um, let's see. All right, so let's close out so you can follow, uh, you can follow the show on Twitter at, uh, this week in Health It, and you can check out our web, web website at this week in health it.com.

Uh, catch the videos on the use YouTube channel this week, uh, in health it.com/video. Uh, please come back every Friday for more, more news information and commentary from industry influencers. We. Lineup, uh, of people over the next, uh, four, four or five weeks. And we're starting to set our, uh, fall schedule as well.

So you'll wanna check us out every Friday. Uh, that's all for now.

​