Newsday: AWS Outage Exploration and Removing Risk from Your Data with Vik Patel

This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong. This episode is brought to you by TIDO. TIDO provides managed integration , EHR, data migration archiving, workflow automation, and 24x7 monitoring to ensure seamless data exchange and improved operational efficiency. Dealing with capacity challenges in data or integration projects and operational optimization issues, leverage TIDO's AI driven solutions to reduce costs and enhance patient care continuity. Ready to elevate your healthcare IT strategy? Visit ThisWeekHealth. com slash TIDO today and discover how : innovative solutions can transform your organization I'm Bill Russell, creator of this week Health, where our mission is to transform healthcare one connection at a time. Welcome to Newsday, breaking Down the Health it headlines that matter most. Let's jump into the news. Bill Russell: Alright. It is Newsday and we are excited today. We've got the the normal crew here and Vik [00:01:00] Patel. How's it going, Vik Patel: Vik? I'm good. It's it's Diwali today, by the way. Happy Diwali. Oh, happy Diwali. Bill Russell: And obviously Drex, the Ford is here otherwise known as uncle Drex as we call him. Oh, don't do it. It's just gonna continue. And then Sarah Richardson back from a, an amazing summit 2 29 summit out in Napa. So it's good to see you back in town and ready to go. And hopefully we won't have a a cad appearance on the show again. Sarah Richardson: They're all asleep right now. Bill Russell: They're all asleep. So today because people are gonna know when we recorded this based on this. So, AWS experienced a major outage that began at let's see, 3:11 AM Eastern time affecting services across the us east, one region in northern Virginia. The outage affected over 500 companies and generated 6.5 million reports on down detector across more than 1000 sites worldwide. Wow. This is, you know, this is what happens when we [00:02:00] consolidate all this stuff. I mean, there's a lot of things running over here. Root cause let's see, this NBC news, I don't know. We'll see how close we feel this one. Vik Patel: Are they blaming? DNS. Bill Russell: While the issue originated from within the EC2 internal network and impacted core AWS services, including Dynamo Db SQS and Amazon Connect, AWS identified the root cause as underlying internal subsystem responsible for monitoring the health of the network. Load balancers. Yeah, it could be DNS. That's a pretty, I don't know, drex. That feels like a pretty broad. Drex DeFord: Broad Bill Russell: statement. Drex DeFord: My gut tells me like somebody probably did something that they weren't supposed to do. Somebody clicked a button from on to off or off to on, and there were a lot of unintended consequences and it took a while to figure it out. This is probably like a change control thing that didn't work the way it was supposed to. Bill Russell: All right. So major services hit included Snapchat Roblox, Fortnite. Oh my gosh. Fortnite. ISS down. Fortnite. Oh God. [00:03:00] Ring single Coinbase, Robinhood Zoom. Duolingo, Starbucks, McDonald's, United Airlines, Delta. I'm trying to see if there's any healthcare. I'm sure healthcare was impacted, but I don't none of those Vik Patel: lot of people screaming were the Roblox and Fortnite users, obviously, you know, they're everywhere. You know, if you go on Reddit or anywhere else, even Reddit was affected, Sarah Richardson: you're hosting, I mean, it could have, it could affect. Claims processing your data collection, insurance verification. Just some of the critical infrastructure, although I had to, you know, I had to go down my little path of, well, if it happened in Virginia, is it not? Maybe it's tied to the government shutdown. . Bill Russell: Here's what it says for current status. Well, AWS initially said the. DNS issue. So there you go. DNS issue was fully mitigated at 2:24 AM Pacific Daylight Time. Customers continue experiencing increased error rates when launching new EC2 instances as of 1:30 PM Eastern time. AWS reported seeing early signs. You know, one of the things I love about [00:04:00] this, I'm trying to figure out what question I want to throw out to you guys. One of the things I love about an AWS outage is. We have more information when an, when AWS goes down than we do in our internal networks. I mean, like, they are very transparent and they immediately put the information out. They put it out. So, I mean, unless it's obviously a cyber attack of some kind, but they generally will let you know exactly what's going on and not just their clients. They let the world though. I mean there's no, if you go to Drex DeFord: health.aws.amazon.com, you will see every outage that they have sort of like a running live dashboard that they update with all the stuff that's going on. So Drex, do you know any health system that has that? No, I do. I do not. I mean it's kind of interesting. There might be some version of this where they send out a regular email update. But yeah, I dunno about the dashboard. Yeah, Bill Russell: what does healthcare take away from this? We are consolidating an awful lot of activity. In fact, I was with a health system that has moved their epic [00:05:00] instance to AWS just this past weekend we were talking about that. So people have chosen to take Epic to AWS They've chosen to take it to Azure. Some of 'em are taking it to Rackspace, others are taking it to other third parties. Some of 'em are hosting with Epic themselves, which which is an option as well. You know what do we take away from this If you are a healthcare leader, Sarah, I'll start with you and then Vik, I'll come over to you. Sarah if you're a leader, because you were moving an awful lot of your stuff to the cloud. Sarah Richardson: We actually, we moved everything to AWS, we did it two ways though. We did the multi-region AWS, which isn't completely soundproof. I mean, you've gotta think about multi-cloud as much as multi-region. However, AWS does have data centers all over the world, and so when we did a full scale move to AWS activity, we did multi-region as one of our primaries, and we did have some backup services with Azure. So the multi. Tenant within the regions and potentially if you can afford it, and your infrastructure and architecture allows for it, the ability to have a backup [00:06:00] cloud provider as well. Bill Russell: So this is one of those no dust statements, but architecture still matters even if you're going to the cloud. Like you can't just go, well, we went to AWS and you know, clearly they have all this stuff worked out. I mean, it's even more so you have to think about it. Sarah Richardson: It's very specifically, and depending how you have it, how you have it designed, that architectural vulnerability like AWS is architected with thousands literally of microservices that are communicating, I think continuously with one another. So when you have a foundational serVike, like we had a lot in their Dynamo DB and the Kinesis experience had problems in the past that can propagate to other services. And so you need to have an API gateway that checks NMO for configuration. Lambda function needs to write logs. CloudWatch has to have database storage for metrics. All those dependencies create really a web where a single, I think failures can trigger cascading problems, but you have to know where all those pieces connect. That's why it's so much harder to be in this business today. It's not a point solution's. One of the things that Vik is so good at [00:07:00] is like I know about things happening before you do because it's not one connection, it's hundreds. Of connections and some of them you don't have control over all of 'em. You only have the awareness factor. You've gotta know where to shut it off as much as you need to know where to connect it and be able to turn it on effectively. Bill Russell: Vik what's your takeaway? Healthcare leaders, what's, what should their takeaway be from this outage? Well, Vik Patel: first of all, Sarah, like do, are you on the Amazon AWS support team? Like, Sarah Richardson: no. But when I went to their services, I had to make sure I knew what all of those, you know Vik Patel: a lot about it. I mean, that's amazing. But yeah, you know who's who's kind of laughing today, it's probably the people who are like, I am not going to the cloud. The on-prem people, you know, who are, who have been like, ah, I'm not still sold on it. which is. Again, that's a whole another story. I don't think it should be that way. But I think could a hybrid approach. Be, you know, what's needed, right? So even for example, in integration, in architecture as Bill you said, one of the [00:08:00] things that we have been talking about is hybrid architecture many times, because let's say you have all these point of care deVikes, right? Like the lab instruments and other vital signs, instruments that you need by the bad side, you don't need all those going to the cloud. And then coming back, right? Like you, you probably need an architecture which supports a lot of that integration. I'm talking about integration in this case, connecting to something OnPrem instead of making that hop every time to the cloud and back because in this kind of situation. If that happens now, you just affected patient care within the hospital, right? Like, that's terrible. That was a terrible architecture. So those are the kind of things that I think you should pay a lot of attention to. So, I mean, yeah, every time this kind of event happens, I think it's a, it's an amazing opportunity to sit back and look at what's in place. You know, is our architecture, the right architecture, and what [00:09:00] are the changes that we may need to make here? Bill Russell: Drex, you know, there's so many things we're supposed to take a look at and I'm not sure we're looking at all those things. I like the architect concept. When we were moving to the cloud and now we were. Doing this in 2013 when we were moving to the cloud, we couldn't get BAA sign and stuff. So we walked very cautiously into the cloud and we thought a lot about architecture. One of the things we did is our failover was on-prem, so our, we moved things to the cloud, but our failover was to fail back. And it is almost reverse Dr if you will. And in, in California it makes sense. because if we had. Our disaster was an earthquake and it could take out, I mean, it could take out everything quite frankly. And we were also serving Northern California and West Texas from there as well. And you didn't want to take them out just because Southern California got taken out. So architecture really does matter. And I'm wondering, you know, as we look at more and more things being moved to the cloud [00:10:00] do we have the skills to do that internally for healthcare? And it's not only infrastructure, it's also security architecture. I mean, do we know what we're doing? Drex DeFord: I think there's this interesting sort of, path that we've been on through the pandemic, where we pushed a lot of stuff out of the data center and into software as a service And we don't really understand what's happening with the software as a service company and who they're using as fourth and fifth level, you know, service providers like AWS or anybody else. And so. It's hard for us, and I mean, part of the reason we did that is that we didn't wanna be in that business, so we moved it out to someone else. I think the same thing when we move workloads, we do other things with companies like AWS. Part of that is that we don't wanna be in that business anymore. I think it requires, yep. Good architecture. That's another whole conversation though, right? Just because you can go from the alaris pump to the cloud and then back or whatever it turns out to be, doesn't mean you should. The technology used to be such [00:11:00] that we built stuff in-house in data centers that had to be on-prem because we couldn't stand the network latency that it took to have it anywhere else. And now the world has changed. and Network latency is way better, and we've got technology that can handle that, and we wind up in this word situation that we do things because we can, not because we necessarily should. The same thing applies to those contracts. When you read those contracts and you look at architecture, make sure you've got penalties in there for downtime and all those things. The problem is that when you're down, you may have the penalties written in, but it doesn't really help you in the heat of the moment when. Your EHR is offline for six hours because something's happening at AWS. I mean, we try to shift the responsibility somewhere else and we try to contractually obligate the risk to someone else, but I don't know if you can really do that. Vik Patel: But, you know, I mean, so just a little bit of counterpoint to that in terms of, being [00:12:00] able to do everything on prem today. I don't think it's I mean, it is possible, but if you have millions of dollars sitting around, right? So in one of my conversations recently with the CIO, you know, we are talking about using Azure Open AI and Databricks pipelines and everything in the cloud, right? And he's like, Hey, why don't I talk to somebody, I think we can do this on prem. And I was like, really? I mean. If you have $500 million, you know, to build the same infrastructure so I think that's where, you know, once I kind of walked him through what it would actually take and if Microsoft would even allow that kind of licensing, first of all, right? Like the Azure licensing on-prem. I mean, I don't even, I haven't heard anybody do that. You gotta build the scale Drex DeFord: and everything too, to be able to keep, you know, you've got a lot of stuff sitting there idling locally. It's part of the advantage of the cloud is that you have, that you can surge when you need to surge. And Vik Patel: so I just feel like, yes, you kind of [00:13:00] almost have to go to the cloud for those things and you know, you just don't have compute sitting around and the right infrastructure sitting around and the scalability, like you just said, drax. So I think some things you will have to go to the cloud. So that's where, again, going back to the architecture, you know, what are the things that you can still get away being OnPrem and should help you with patient care, but the ones where you do need to use the cloud. How do you do it intelligently? So if when something like this happens, you're still not impacting patient care. Bill Russell: So Vik let's get into your world. So Tido really focuses in on the data aspect of well data in an AI world, data in a cloud world, if you will. What do we need to be thinking about when we're moving to the cloud and our data and being able to, I don't know, do what we want to do? I'm gonna leave this really vague in general. Sure. Do what we need to do with data. I mean, I hear stories all the time of people are like, [00:14:00] oh man, our data is o over in this cloud data center and over in this cloud. data center And like, we didn't think it was a problem, but now that we're doing. Ai, we have to bring all this stuff together or some workflow where they're bringing it all from all over the place. You know, I what does data in the cloud world and the AI world look like? Vik Patel: No, that's a great question and I'm glad you kind of capped it a little bit vague. So, I will go to one of our latest solutions around Tito Cortex ai where. We are using our expertise that we have built in-house with our Mitre ai, the proactive monitoring of interfaces and applications. So we are partnering with organizations to help them implement their AI applications. Right? Like we are not trying to be Emmy or whatever, you know, all the other products that, that Epic and the other EHRs are doing. That's not what we are trying to do. We are just partnering with the organizations, bring our data scientists, bring our technical team and work with them. And [00:15:00] you know, that's where using the data, first of all, you need to have the data in the right format, right? So for example, a lot of times for the AI solutions, it does make sense in most cases, to have it in the FIHR JSON format, for example, it is able most, you know, and then you can expose those as APIs and then you can also expose that to the AI applications. To the analytics, all that stuff, but sending data to the cloud, it's like, oh yeah, let's just send all our EHR data, you know, to the cloud for processing and then maybe use it in open ai, which would be terrible if you just send PHI in there. So one of the things that we are actually doing is we are working with organi with three organizations today where. We are helping them again with the architecture and how this would look like and how to expose that data correctly to all these different applications that they want to use from a clinical standpoint, operations standpoint. And one [00:16:00] of the ways I think could be really useful for organizations is to de-identify the data, right? So you can still have all the data that many of these applications, especially the analytics and the reporting side of things that we'll need. They don't necessarily need to know who that John Doe is. You know, you need to have the data that you can use to make sense of it and have some actions on top of it. But why not use de-identified data? Right. So I think that could reduce the risk largely. And you know, we are creating, a pipeline to actually do that on-prem or wherever they have their data before it goes to Databricks or before it goes to the cloud and blobs. And then, you know, that's where it's then formatted for the AI data lake. But I would suggest that, I would say don't expose all that data. I mean, that's taking on a lot of risk once it's out there. And once it's consumed by some of these applications, you have no idea where it's going to end up. Bill Russell: So Drex, you just [00:17:00] hosted a chief Information Security Officer event. And he said the magic word, he said risk. And as he's sort of describing that, I'm sure that was part of the conversation you guys had. Drex DeFord: There's so much to that, right? Like are you gonna build your own LLM model? Are you going to try to leverage an existing LLM model like OpenAI and then. There's obviously levels of risk in that conversation around that first decision. And then how are you going to manage the data that goes into that Part of it certainly ties back to, I mean, I think when we go back to the. Fundamentals. The fundamentals are, do I know what data I have and where it is, and am I sure that it's actually correct and clean and usable and, you know, all of the things that go along with the data conversation. I think a lot of health systems still kind of struggle with those. Sort of fundamentals in a lot of ways. They may be able to get clean data out of one system or two systems or five systems, but that's not necessarily what they need to make some of the decisions that they're making today. [00:18:00] So they get sort of half the data through an LLM model through some analytics, and then they sort of trust their gut on the rest of the stuff that they wish they had data for. Vik when you're. Talking to health systems who are kind of in that position, how do you help walk them through that or think through that challenge? Because I think for the health systems that were in my CISO event, there, a lot of them are still struggling with that issue. Vik Patel: Oh yeah. And that's where patient matching becomes a huge issue too. Right. So as you were saying that, I was thinking about a recent project where, you know our the data team as they were converting a lot of this data. Four use in the cloud. Like yeah, they had to like, make sense of, hey, is this John Doe coming out of this system the same coming out of this other eHR. But one of 'em had terrible data, right? Where it, it wasn't very easy to do the matching and we had to update I think they had to go in and update [00:19:00] the algorithm in terms of the trust factor where then you have to say no. You know what? We trust this data here more. So when it comes to matching the two people, I'm just gonna use that example. You know, there's all the other clinical data too, but just to kind of keep it simplified, that's where you may have to adjust how you make use of data, which data you trust more, and then allocate more points to the more trusted system. So you have to make those kind of adjustments, but having, you know, you just need the right team. Who can understand that and make those adjustments. So I think that's a great example. Drex in terms of, yeah, like that's part of the process when you are putting all the data together that your future AI applications will use. Drex DeFord: Yeah, I think that's really interesting, right? This idea that because we have so much crappy data in so many different systems that maybe AI can help us sort that out. Maybe that's the great use for ai, Bill Russell: the use case. It's really interesting. I mean, Sarah, what are your [00:20:00] thoughts on just data and our data architecture. I mean, you guys your last organization, you guys were not only just moving everything to the cloud. I mean, you were really rethinking how data was gonna be used in that business and how it was going to be exposed and how it was going to be handled. Just across the board you had great data pipelines and whatnot. I'd love for you to weigh in here. Sarah Richardson: Yeah, the what was most important was getting the, just the foundational team in place. So we had the whole like data fabric, so that was just a title or ti something people called it, but higher leaders who knew how to do data, people who could do architecture, who could do the governance, who could do the reporting, the intake, the normalization. That, well, we couldn't de-identify it for married reasons, the way we were structured with the health plans, but creating a space and only allowed for that and all be a clean space. Like we had to create owners of every single pipeline. People knew where things went, and then we did work with the health plans to be able to pull their own reports. I mean, anyone who's gone through a complete data think restructuring in the organization [00:21:00] realizes you have probably thousands of orphaned reports out there. So that whole first year was getting buy-in for the program as much as it was cleaning up the backend and then having to retool and upskill the teams, and then having to work with the partners to want to utilize their own data from a self-serVike analytics part as much as us feeding that information out. The hardest part of all of that. Wasn't actually the architecture and the technology around it. It was getting the CFO and the business owners to agree to the cost of ongoing care and feeding and maintenance. It's not like you clean up your data once and it's done, and then you go ahead and just have this beautiful warehouse with all these reporting features on top of it. All that lineage and quality and tracing that information is really key. In addition to the security aspect of something goes wrong, what needs to get turned off first, and so that was an entire team. I always went back to the board and to my C-suite peers to explain not just the efficacy and value of the program, but why we had to constantly fund it and what that meant for us organizationally. And so I think for anybody who has a really robust [00:22:00] data program and has that kind of like democracy within their organization, it's what is it actually worth and what are you willing to do to keep it up and running? Because otherwise you're just doing a cleanup job every couple of years and then all the risk of even quality and outcomes from that can be pretty persnickety. So. Hire people that is their job. Bill Russell: When I came into healthcare again, 20 10, 20 11 ish timeframe, one of the first things I did was stand up, get data governance program. And because every other industry I'd ever been in had a data governance program, I just assumed that healthcare would have a data governance program. And then the thing I think that still surprises me here we are a little less than a decade later. Or over a decade later, and we still hear people saying, well, you know, we're gonna stand up a data governance program. I'm like. Really? Wow. Like, how can that still be a thing that would be like saying, we're thinking of hiring our first ciso. We would all look at 'em like, are you kidding? But on the data side, we're like, oh good job. I'm like, no, I mean, [00:23:00] look, by all means get started. Get that data governance in place. And I think what Sarah you're saying is it's not a one time event's it is a program it's something that, you know, gets better with time and you have to define your data. With direct on the thing. I have to say, you know, if you de decide to, you know, what is your risk profile look like? How are you gonna, you know, data at rest data you know, you get encrypted at rest data and transport and all those things. You gotta build all that stuff out, which. I'm sure most health systems have done 'cause they have a good security program. But on the data side, you know, data definitions make all the difference in the world. I mean, let's just take the basic most simplest thing and most health organizations have taken care of this, but when I came in, I remember the first request I got was from the reporting team and they're like, Hey, we need five more people. And I wanted to be the good CIO. And I'm like, oh yeah, of course you need five more people. Let's get you five more people. And then the next year they came back and said, Hey, we need five more people. Until I, you know, and at that point, of course, you don't have to do this to me twice, then I'd sit there and go. [00:24:00] Okay, let's have a conversation. And what I realized is the request for reports was gonna be endless. It's insatiable. And the reason is, 'cause we never got in front of it and we never identified, you know, what reports are actually necessary. Well, first of all, what reports are actually getting used? That was my first report. That was like. Are you kidding? None of these reports are getting used. Drex DeFord: Why are we doing, we have 25,000 reports in the inventory and 22 of them are actually used on a monthly basis or something. Yeah, Bill Russell: we had a great conversation around simplicity as a strategy. For it, like, just as a core design principle for it. And in our CIO meeting, it was really interesting to hear. It's like, look, complexity is it is sort of a gravitational pool in healthcare and part of the role of the CIO I'd love, we'll close on this. I'd love for each of you to sort of. Discuss this from your own perspective. The gravitational pool is towards complexity. It's towards more applications. It's towards [00:25:00] more interfaces, it's towards more reports. It's towards you know, more cloud vendors. It's towards more, and everything I'm saying is just more complexity. And so how do you hold the line? And it is simplicity, a good. Target, is it a good value that we should be shooting for? I'm trying to figure out who I want to give the last word to. Vik. I'm gonna give you the last word. So Sarah, we'll start with you. Sarah Richardson: I will put this on my banner that I fly behind a plane or anywhere else. To me, a hallmark of how well we are functioning as an IT organization is when other people inside our organization can tell our story on our behalf because we are there to serve others. And so if you're the chief financial officer. You can tell the quote unquote it story in terms of how it's benefiting your organization. We're doing the right things. Drex DeFord: You've heard me go off about this before, but I mean, I think it has to be an anchor in all the things that you do, I mean, if you can have a simple, you know, Maslow's hierarchy of needs, if you can have a simple architecture, a simple infrastructure, if you can [00:26:00] have a simple group of applications it helps you understand all the things that you need to know about your data so that you can. Report and make decisions on data because the environment is simple, it's easier to secure because the environment is simple. It's more likely to have uptime and ha not have some crazy downtime that you can't figure out, and you have to talk about it as a DNS outage because the environment is simple and so. The simpler you can make everything and you're right. I mean, I think the environment pulls you toward complexity every day. And part of the job of the CIO is to figure out, to be the champion of simplicity that has to be part of the job. Bill Russell: Vik, last word on this topic. Vik Patel: Yeah I think simplicity is actually very difficult, right? Amen. Absolutely. It's difficult because if you are making a product that's very simple, let's say, you know, everyone uses the example of Ubers, but [00:27:00] behind the scenes, you know, it's probably very complex, but it's simple for the people who need to use it and I think it should be similarly in healthcare, it should be very simple. For the care teams. Right. It should be like no brainer to them. It should be, like we don't, they shouldn't be calling into the help desk when there's an issue. We should know about it and I'll, you know, talk about how our mi ai does that very proactively finds things. But I think that's the difficult part, like simplicity. If it was that easy, everybody would've done it. So I think, again, I don't think it matters how many applications at the end of the day, you can have one application, but you can still make a mess of it still won't be simple. So I, you know, from that, Bill Russell: where you were going, I wasn't gonna agree with you, but from that perspective, I agree with you. Even with one application, it can get messy. Vik Patel: Like as Sarah said, having the team. On the same page, like everybody kind of getting together [00:28:00] and coming up with a solution that everybody understands. Again, probably not gonna happen in one day, but just take your time and architect it the right way. Bill Russell: this is all real simple to me. It is. You know, the only reason we need data is to answer questions. Right. That's what we needed. So people are asking questions they need data. And my experience this morning was, I need to know how many partners were gonna be at our city or Philadelphia City Staff Roundtable next week. And so I started it in, in Google Sheets and it wasn't there. Then I went over to Airtable and it wasn't there. It probably was in both of those locations. I just couldn't find it there. And then I and then I ended up looking at a different Google sheet and found it over there and. How that experience probably plays over and over again in every organization of, they just want answers to questions. And it's one of the reasons I think that people love generative AI much more than Google search. They just want the answer to the question. I don't want [00:29:00] 50 websites to go to. I just want an answer to the question. And now with generative ai, we ask the question and it comes back and it says, Hey, here's the answer. If you want to know where I got the answer. It's here, and here. It's these 10 websites that I went through. And so, you know, again, I, maybe I'm oversimplifying, but data is about get, getting answers. And at the end of the day the best way to do that is to know what your data is. What questions it can answer and be able to find it and funnel it into the, in the right direction. I think that's what you guys are doing. Vik and I think we're in wild agreement, which our production team is not gonna like, they like it when we argue and disagree with each other, but I think in this one we're pretty well in lockstep with each other. Hey Vik, thank you for coming on the show. It's always a pleasure to hang out and discuss the news with you. Vik Patel: Thank you so much for having me. That's Newsday. Stay informed between episodes with our Daily Insights email. And remember, every healthcare leader needs a community they can lean on and learn from. [00:30:00] Subscribe at this week, health.com/subscribe. Thanks for listening. That's all for now.