50 Agents on Paper. An Audit Finds 76. | Newsday with 229Project

This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

50 Agents on Paper. An Audit Finds 76. | Newsday with 229Project

[00:00:00] Uh, I'm not sure we've ever brought software in before that we're like, well, this might get out and wreak havoc, but welcome into the uh, welcome into the kingdom.
[00:00:12] [music] All right, it is Tuesday and we are strewn around the around the country. Although, Drex, you're back. You're back from the Backstreet Boys concert. We want to hear about that. Sarah Richardson, you're in Nashville tonight and uh I am in Florida. My next trip is to UGM. So, but when they hear this, UGM will be going on. This will be aired on
[00:00:38] that Monday. So, uh let's get let's get some quick updates. Uh Drex, uh Black Hat, how was Black Hat? Uh it was, uh you know, it was good. Um, I only went in for one, actually [clears throat] two days, but just a quick trip. Um, one night I went to the world premiere of Midnight at the War Room. Uh, movie is produced by a company called Seus. Uh, great movie. Amazingly
[00:01:06] enough, sat right in the middle of four or five of my friends, all who wound up being in the movie. They had parts in the movie. It's a documentary. They just talk about everything from uh you know water systems being hacked, hospitals being hacked, um the the the process that goes on in the background when these bad things happen. Um national sort of figures, local figures. It was
[00:01:31] great. Um really amazing hour and a half. Totally everybody should see it. I don't know what the plan is for distribution. I know they're even talking about doing a series. So really um super excited and that was that was Is this the kind of movie I'm going to want to take my wife to?
[00:01:47] Uh she'd probably love it actually. It'd be really interesting I think for uh [laughter] I think you need to spend more time with my wife. You think she would love it. She's a documentary person. She uh Yeah, but not on cyber security. I think she she would space out. Although we did see uh we did see the Odyssey which was uh interesting. The two of us won her her pick. So, I haven't seen that. How was it?
[00:02:12] Uh, you know what? Here's here's what I tell people. If you are if you read the book and you are a fan of the book, you're not going to like it cuz it's not as true to whatever. But if you're like me and you didn't read the book cuz you you're a slacker in in high school, then uh it was really good. It was it was very entertaining movie. So, it's uh it's, you know, it's one of those like I'm I'm not a purist when it comes to
[00:02:38] has to follow the book, you know, perfectly. So, I thought it was I thought it was really entertaining. The the other uh speaking of entertainment, I went to Backstreet Boys concert at the Sphere. Um Elicity had two suites. A ton of cyber security folks uh were in those suites uh during Black Hat and it was uh great fun. I'm not really a huge Backstreet Boy fan, but man, I was kind of blown away by how many of the songs I actually just knew
[00:03:03] through Osmosis. It was uh it was a lot of fun. And Sarah, you've been all over. So, you're uh Charlottesville and then you took what a train down to uh to uh Nashville. Uh planes, trains, and automobiles.
[00:03:18] No, I didn't take a train. I I drove a rental car. [laughter] Oh, you drove a rental car. Visit visiting friends. Tell me about Charlottesville. I mean, what what uh what were the topics there?
[00:03:29] Yeah, Charlottesville was absolutely incredible. So, the first piece I loved and I got to fly into Richmond Monday. I stayed with a friend um which is great. It's one of my girlfriends I've known for a few years. We'd never met in person. So, we got to do that which was awesome. Um which you know our universe today we meet so many people when you get to meet them in person you're like yes I definitely want to hang out. Uh
[00:03:48] then we did the dinner in Charlottesville as Jarus and I and I'll start at the end. Dave Matthews lives there and he was at the bar as we were walking out. So, we did geek out and try to get an autograph, but it was still kind of a good check mark on a night that everybody showed up. Partners were super involved and partners we knew and always enjoyed spending time with. And it's the conversation off the charts good. I mean, you literally had, you
[00:04:08] know, Brett McMillan, you had Andy Marowski and Shannon Becker and so many others who just helped us fill a room. Barbara Ballard like great conversation, great perspectives, everything from interoperability still being a thing and the ability to do appropriate integrations to how well are our pilots proving out in some of the AI spaces and what are we doing that's not even inclusive of AI? What
[00:04:32] are the things that are still keeping people awake at night or things that people are focusing on or thinking about? Tons of energy around the protection, the cyber side of the governance and understanding of AI. So, we used to talk about AI governance in terms of like what are we deploying?
[00:04:48] Now, it's okay, what cats are in the bag? How do we keep them from getting out of there? And what are the resilience mechanisms we have from a continuity perspective if you don't even know what happens inside of that black box? How the heck are you supposed to have a downtime procedure for it?
[00:05:01] Uh, good point. Um, yeah, on that same night that you were there, I was in uh beautiful um Raleigh Durham, North Carolina. And uh you know what? really good. We had uh we had some interesting conversations probably along the same lines you did, I would say. You know, we did talk cyber and resilience. We spent a lot of time talking about uh about AI uh for sure.
[00:05:25] Um ROI models around AI, governance around AI, um prioritization, capacity, uh those kinds of things. Application rationalization came up a couple times.
[00:05:37] It's uh amazing how hard application rationalization is. It just it's hard it's hard to me believe we're still we keep talking about it. I mean we're talking about not not that it's it's a thing that you got to keep doing all the time but uh like people just feel like it's so hard politically.
[00:05:54] Nobody is better at it than Pete Marx at Wake Med. I mean he is he's he's vigilant. He is determined. He has a a uh a narrative around it. It makes perfect sense. And in his words, you know, he goes, "We fought hard to reduce 20 applications." And you know what we did last year? We increased. It's [laughter] like it just overall they went up.
[00:06:18] Yeah. Cuz they just keep coming. It's like, can how do we how do we stop this? It's like it's just really it takes vigilance. And um I don't there's a part of the conversation that we're just not getting through yet. And I think it's it's it's the it's the a word. It's architecture. It's like look every time you add an application it increases complexity, increases cost, increases support and staff and it increases all
[00:06:45] those things and you know you can make an individual case and this is what happens in healthcare. You make an individual case on this single app. It's like look, it helps radiology by this and scheduling by this and it's going to save us this much money. And it's great.
[00:06:59] And in of itself, if you treat each one of those cases individually, you go, yes, all right, that's good for the health system. But when you layer like a thousand of those on top of it and it's, you know, then all of a sudden it's like, well, why do we need so many IT people? Why do we need such a budget?
[00:07:15] Why do we need to support all these things? It's like well because one doesn't increase the number of people but a thousand increases it by you know 100x the number of people it requires to support that support conversation continues to come up. I have a feeling uh when this airs I will be sitting in the CIO forum at Epic. I have a feeling that will be a uh a significant part of the conversation is how do we reduce the
[00:07:40] cost of um staff and support of these EHR systems. They're big. They require a lot of people. Nobody's nobody's arguing that. But the the number of people cannot go up. It has to go down.
[00:07:56] It's a it's a pretty hefty uh pretty hefty lift. Um, and so anyway, those were those were some of the conversations. Uh, Sarah, I want to get to your topic, but first, Drax, I want to talk to you about this. Um, it seems like every frontier model in the last 3 weeks has um shared that their model got out of the lab. And uh, I wonder how how
[00:08:24] much of a concern is this for health systems? Is it a concern? I mean, it it's almost like a right of passage. It's like, hey, our our model's so good, we can't contain it. I'm like, hey, that's not good. Like, I know it is. It is one of those things like, is it a downside? Is a is it a is it an some weird attempt at like a marketing sales pitch is so good it keeps getting out of the sandbox. I I mean, for health systems, I think just
[00:08:50] for everyone in general, as everyone knows, I'm a little paranoid about a lot of these things. And so when I look at it, I think about like the good old bad days, good old bad old days of W to cry, right? The Russians were attacking the Ukrainians and they put a virus in the banking system, I think, to try to take down the banking system. Turns out the Ukrainian systems were connected to the rest of the world systems and suddenly
[00:09:16] within four or five hours, North America was going down. Completely consequences. I think there's this other problem right of uh you're taking these models especially the open models and you're running them on your own infrastructure.
[00:09:31] So some companies just doing this on their own. They're creating a sandbox. They're doing some kind of a tests. I worry about that. Not just the model getting out and attacking two or three companies. uh maybe looking for some weird things like you know the answers to a test but actually getting out and doing something bad to the entire internet. Then we already know that
[00:09:54] they're doing things like breaking out uh multiple agents and then setting up chat rooms and exchanging uh information between the agents uh about how to hack and how to social engineer individuals to get them to load malware and all this kind of stuff. So we're there. It's it's a little it's a little scary to me.
[00:10:15] Well, this is the part that concerns me. It's like Anthropic saying we put it in our sandbox, which I assume is a better sandbox than the one I'm going to build. Like, oh, and it got out and it started doing crazy stuff. Open AI saying the same. They're all saying the same things like Chinese models.
[00:10:30] I'm bringing I'm bringing some of these things inside, you know, it's the Trojan horse. Speaking of the Odyssey, it's like we're bringing it inside the four walls.
[00:10:40] Yeah. like and if if I can't contain it, if I can't control it, uh I'm not sure we've ever brought software in before that we were like, well, this might get out and wreak havoc, but welcome into the uh welcome into the kingdom.
[00:10:54] It's a whole new world and it just keeps getting weirder. So, when we talk about it is who's responsible when a diagnosis is signed off on by a physician, but it was generated by AI. Obviously the doctor's the human in the loop and the liability still lies with the human being. You build an agent or you create a code and it somehow gets out for reasons let's just assume good intent beyond your control. Who then is accountable for
[00:11:19] that model automating itself out of the the instructions that it's been given? I I wrote about this in two-minute drill extra uh uh yesterday on Sunday and um I mean it's just the strange situation that if it was a person obviously that person probably would be investigated, arrested, probably would be fired from the company, might have to go to trial
[00:11:44] and you know a whole bunch of other things. But you know since it's an agent it seems like the answer is oh my goodness my agent got out of the sandbox. uh you know there's no there doesn't seem to be at this point any real impact on the company from which the agents escape. Now, I would say Open AI is probably paying, you know, there there's probably some negative financial
[00:12:08] impact of this happening to them, and we just don't know what it is because it's not public. But, uh, but still, yeah, I I I think that's that's a question we're going to continue to struggle with, not only for clinical operations or business operations, but just in general, when the model goes wrong, who's responsible for has not been set yet on any of this stuff. So, it's going to be it's going to be
[00:12:33] Yep. Uh Sarah, your topic for today. Yeah. So heading into a big conference around a major EHR is a couple things that the integration perspective like integrating some of these AI opportunities is almost like a new technical debt crisis. It's one of the things that the CIOS and others have been sharing. It's, you know, a who owns is this the CIO, the CMIO, the chief AI officer of clinical operations, who owns
[00:12:58] this this governance model and if it's shared, but truly the integration ability and trusting the EHR to be able to do the right thing with some of these integrations outside of their walls becomes a much tougher conversation. So, as we think about healthcare leaders being rewarded for launching the AI pilots, maybe it's the ones that produce the sustained measurable outcomes at scale, knowing what to retire early when
[00:13:22] they do. That's always a big conversation in our events. But really, the technology problem starts becoming a bit of leadership and a bit of the infrastructure because it is hard to integrate some of these models and also it's harder to pull them back out even once you've had a pilot that goes into play. But launching a pilot is not success. It's the scalability and is it replacing one or two other things because you've already optimized other
[00:13:47] solutions that you have in house. Like the minutia of doing this really really well is almost an untenable scale in and of itself because of the number of things that are out there. And if you ask someone today, how many agents do you have deployed in your organization?
[00:14:01] They might tell you 50. An audit turns up 76. And what are they all doing? But the ability to integrate effectively and still be able to scale it in a way that's meaningful to the organization while controlling some of those costs and expecting the right outcomes that still to me is really hey AI works. We know it works. We use a little bit of everywhere. But our ability to operate it effectively inside a health care
[00:14:24] system has not yet caught up. This is the question I've been asking. I asked this last week in in Raleigh. I'm like all right so uh talk to me about trust, right? So majority of AI is going to come into healthcare through our existing partners. It's going to come in through through Workday, through Service Now, through our EHR partner, through uh potentially our imaging partner, through, you know, fill in the blank.
[00:14:48] That's how that's how a majority of it is going to come in. And my question to them was sort of like, okay, so if it comes in from Workday versus Epic, do you trust it more or do you trust it more if it comes in the other direction?
[00:15:03] What kind of testing are you putting around it? Do you have the tools? Do you have the skills to put it through the testing? Do you understand the you know the the decay that happens around it? Do you understand the harness that happens?
[00:15:16] Do you understand? And uh which got us to another question which was okay. Um and that in of itself is an interesting conversation that we had but then I I took it to all right how much knowledge do you need to have and this is predominantly it was predominantly CIOS I think it's six six of the 15 people there were CIOS uh couple uh two CTOs uh couple SISOs and whatnot but my question was how much
[00:15:44] AI knowledge do you need to have because part of me is like all right so you set up this governance group who does the technical certification that this tool is what it says it is and does what it says it will do. Now I understand clinical goes to clinical and you know and and uh compliance has its role and security has its role and whatnot but uh in terms of it actually doing what it
[00:16:09] says it's going to do and uh you being able to protect the organization uh from those things. How much knowledge do you need to have? And that was a that was an interesting conversation. Everybody feels like they're a five, by the way.
[00:16:22] And that's the answer. Note about that. Yeah. On a on a scale of of 0 to 10, that's what you say when it's like, hey, I'm I'm not I'm not a neophite. I know some stuff. But the but you don't you want to communicate you know some stuff, but you want to communicate, oh my gosh, I I know I don't know an awful lot. So, I'm a five. Um, now again, which is the kind of the cop out is what
[00:16:46] you were saying, right? Kind of the Yeah. Yeah. If we were running the 229 health system, the one that we used as our case study for our HCSP program, I mean, granted, if it was the three of us, we probably wouldn't need to hire somebody for AI because we're pretty deep. However, at what point do you know that you need to either contract or hire the type of education or expertise so that you feel comfortable with exactly what you just
[00:17:09] shared? Like what would be what would be the catalyst for you to do that? Or or is the responsibility the partners and you're trusting that the partners going to have done the privacy and transparency and accountability and resilience and trustworthiness that the model works the way it's supposed to? Do do you leave that to the part is that what I'm buying when I buy these?
[00:17:31] Well, let let me give you my my example which has nothing to do with AI. So, I'm I'm doing work on my house and um the I'm getting electric into the this addition that we're putting on and the guy comes out and says, "Hey, you need a 300 amp service pulled to your house." Okay, I don't know electricity, but I do know it has absolutely nothing to do with square footage. He's like, "You have this much square footage. You need
[00:17:53] 300 amps." I'm like, "Uh, last I checked, square footage doesn't use any amperage." Like, it it uses zero amperage. This is a math problem, people. I mean, it's like some kind of a regulation probably that says that if you have this much space, you I have a charger in my garage. I have two dryers. I have it's a math problem.
[00:18:11] And the guy's like, "Oh, no. A house this big needs this much." I'm like, "No." And I literally I'm like, "I'm not working with this guy." And and I, you know, got him out of the house cuz I'm like, it's a math problem. It's how many things that use electricity will be used potentially at the same time. That's what that's what it is. I had the same thing with IT people when I came out at St. Joe's. They were like, "Oh, you
[00:18:33] can't do uh imaging across the way and it has all to be local." And I'm like, "Well, how much how much bandwidth does it use and how much?" And they're like, "Well, it doesn't matter. It just can't be done." I'm like, "It's a math problem, people. Stop it. Like, it's a math problem. How much are you trying to shove across that pipe at what distance and its latency?" And you measure it and you come back and you answer the
[00:18:55] question. You know what happened was when we were done we're like oh we can do this remotely and we moved it to the cloud and we moved it out and it was one of the first organizations that moved to the cloud for imaging and it's like it's a math problem like why do we why do we do this anecdotal stuff all the time but I say this story this time to say I think CIOS need to have enough knowledge to ask really good questions and to know
[00:19:19] if the answers are the right answers and if you don't have at least at skill in your bag then this oh I have a great team great are they going to follow you around in every meeting and ask the right questions are they going to go to the executive meetings with you and ask the right questions you need to spend time with them so that their at least their understanding or their their questions that they would ask come out
[00:19:44] of your mouth and and you can vet that that's that's my take but I'm curious what you guys think I mean is that is that too far cuz I I definitely lean on the you need to be more technical in the CIO role than than most. And I've people have argued with me and people I I respect have argued with me on this point.
[00:20:02] That that's always been the great CIO role, right? the role of translator, translating business needs and clinical needs into technical information that your team can act on and taking technical information and being able to listen to those business and clinical operation stories knowing enough to be able to ask the right questions so that you can you can affect that translation uh well and when you can't do that I'm
[00:20:28] not totally exactly sure what the what the job amounts to because for me that was always like a core critical component of success was being able to to do that translation. You also remind me of like I feel like there's a Chvy Chase movie somewhere or something where uh the electrician or the guy the next door neighbor comes over and asks him like you running 110 into here and he's like yeah 110 111 whatever it takes.
[00:20:54] Mr. Mom, what is it? Okay, Mr. Mom. Oh my gosh. Tuna tuna. Oh my gosh, this is hilarious. Totally [laughter] remember that. I just forget what the 220 221 whatever it takes.
[00:21:05] That's it. [gasps] Oh my goodness. You know the whole thing about being a translator, being a mathematician and all the ability like think about when cyber started becoming a big deal. I remember 20 plus years ago being the person who explained why everybody cared about cyber and smart enough to be able to how did you vet new solutions coming in with a cyber lens when it wasn't you didn't have a security team or a security checklist in
[00:21:28] some cases at that point. We're at that same inflection with AI. And so you got to educate everybody enough and you need to find a couple ninjas who really like live and breathe it so that the smart questions are being not just asked, but someone needs to know to what to do with the answers that are coming forward. And you can't just rely on your partners because all of a sudden you have partner soup and unless they're all figuring
[00:21:51] things out together, you're still left with all of those solutions in one place. and the complexity of their interoperability and integration is a little bit nuts. That's the part where I was so happy to hear Shannon Becker talk about this in Charlottesville. She has an architect for every domain now in the application universe as they think about how they're optimized, utilized, rationalized, you name it. But literally
[00:22:14] was so proud and a couple of others to say, "Yes, we have an enterprise architect that is the starting point." And you love hearing that. And then that enterprise architect clearly having the AI chops that are available, but it's what we've all said for years. Where's your enterprise architect in this conversation? And if you don't have one at this point, you got a whole bunch of other pieces that are being missed separate from whether or not AI gets
[00:22:36] plugged in somewhere. Yeah. And this is what I appreciate about the two of you and and why I like having these conversations is we're all uh learning. Look, we all started like three or four years ago. I mean, we knew of AI, but we all really started we we all sort of had this this gun that went off about three years ago with the chat GBT moment where it's like, oh, we've got to learn this. And you guys, much
[00:23:01] like myself, just dug in and said, "All right, let's learn this." And we're all actually learning different. And it's actually kind of interesting because we're all learning different aspects of it. I'm I'm into deep into the programming. Drax, you're into the cyber security. Sarah, you went full-fledged on, you know, what's it going to take to be a leader in the AI era and went and got training and and education from MIT.
[00:23:23] We can't call you an MIT grad yet, though. We can we can say you you passed that course. But I mean, that's what it I think we call her an MIT grad. I think I mean, let's be honest, I'm a UNLV girl, so take that.
[00:23:33] She's certified. She's certified. Yeah. Certified. Certified or certified? Got all those things. Well, hey, I this is uh the these are exciting times. By this time uh next week when we record, we actually will not be recording on Monday because we're going to be busy, but when we record maybe on Tuesday or Wednesday next week, uh we'll have sat through the uh the epic announcements and there'll be a lot to digest and
[00:23:58] that's probably where we will spend um uh we'll spend next week plus we'll be doing a survey of people who have sat in the room. 229 members will be giving us feedback, their feedback from their perspective and I'm looking forward to that because I think you're going to have perspective that is from large systems and academics and you're going to have perspectives from rural healthcare and critical access hospitals
[00:24:22] and I think sometimes their perspectives a little different. I think that's going to be that's this might be one of my favorite things we've ever done because it's really going to be fresh right out of the gate. You just went through the presentations, you just went through the briefings. what are you thinking? What's next? Like you're going to really get some interesting insights.
[00:24:42] So, if you're going to Epic and you get uh you get an invitation to the survey, take it because you're going to be able to see the feedback from everybody else, too.
[00:24:51] And if you have a place to stay on Tuesday night, we have people who still need rooms. [laughter] It's brutal. It's brutal to get a hotel room up there. Anyway, hey, uh great seeing you guys and uh thanks everybody for tuning in. Thanks for listening.
[00:25:06] That's all for now. [music] Uh, I'm not sure we've ever brought software in before that we're like, well, this might get out and wreak havoc, but welcome into the uh, welcome into the kingdom. [music] All right, it is Tuesday and we are strewn around the around the country. Although, Drex, you're back. You're back from the Backstreet Boys concert. We want to hear about that. Sarah Richardson, you're in Nashville tonight
[00:00:28] and uh I am in Florida. My next trip is to UGM. So, but when they hear this, UGM will be going on. This will be aired on that Monday. So, uh let's get let's get some quick updates. Uh Drex, uh Black Hat, how was Black Hat?
[00:00:46] Uh it was, uh you know, it was good. Um, I only went in for one, actually [clears throat] two days, but just a quick trip. Um, one night I went to the world premiere of Midnight at the War Room. Uh, movie is produced by a company called Seus. Uh, great movie. Amazingly enough, sat right in the middle of four or five of my friends, all who wound up
[00:01:10] being in the movie. They had parts in the movie. It's a documentary. They just talk about everything from uh you know water systems being hacked, hospitals being hacked, um the the the process that goes on in the background when these bad things happen. Um national sort of figures, local figures. It was great. Um really amazing hour and a half. Totally everybody should see it. I
[00:01:37] don't know what the plan is for distribution. I know they're even talking about doing a series. So really um super excited and that was that was Is this the kind of movie I'm going to want to take my wife to?
[00:01:47] Uh she'd probably love it actually. It'd be really interesting I think for uh [laughter] I think you need to spend more time with my wife. You think she would love it. She's a documentary person. She uh Yeah, but not on cyber security. I think she she would space out. Although we did see uh we did see the Odyssey which was uh interesting. The two of us won her her pick. So, I haven't seen that. How was it?
[00:02:12] Uh, you know what? Here's here's what I tell people. If you are if you read the book and you are a fan of the book, you're not going to like it cuz it's not as true to whatever. But if you're like me and you didn't read the book cuz you you're a slacker in in high school, then uh it was really good. It was it was very entertaining movie. So, it's uh it's, you know, it's one of those like I'm I'm not a purist when it comes to
[00:02:38] has to follow the book, you know, perfectly. So, I thought it was I thought it was really entertaining. The the other uh speaking of entertainment, I went to Backstreet Boys concert at the Sphere. Um Elicity had two suites. A ton of cyber security folks uh were in those suites uh during Black Hat and it was uh great fun. I'm not really a huge Backstreet Boy fan, but man, I was kind of blown away by how many of the songs I actually just knew
[00:03:03] through Osmosis. It was uh it was a lot of fun. And Sarah, you've been all over. So, you're uh Charlottesville and then you took what a train down to uh to uh Nashville. Uh planes, trains, and automobiles.
[00:03:18] No, I didn't take a train. I I drove a rental car. [laughter] Oh, you drove a rental car. Visit visiting friends. Tell me about Charlottesville. I mean, what what uh what were the topics there?
[00:03:29] Yeah, Charlottesville was absolutely incredible. So, the first piece I loved and I got to fly into Richmond Monday. I stayed with a friend um which is great. It's one of my girlfriends I've known for a few years. We'd never met in person. So, we got to do that which was awesome. Um which you know our universe today we meet so many people when you get to meet them in person you're like yes I definitely want to hang out. Uh
[00:03:48] then we did the dinner in Charlottesville as Jarus and I and I'll start at the end. Dave Matthews lives there and he was at the bar as we were walking out. So, we did geek out and try to get an autograph, but it was still kind of a good check mark on a night that everybody showed up. Partners were super involved and partners we knew and always enjoyed spending time with. And it's the conversation off the charts good. I mean, you literally had, you
[00:04:08] know, Brett McMillan, you had Andy Marowski and Shannon Becker and so many others who just helped us fill a room. Barbara Ballard like great conversation, great perspectives, everything from interoperability still being a thing and the ability to do appropriate integrations to how well are our pilots proving out in some of the AI spaces and what are we doing that's not even inclusive of AI? What
[00:04:32] are the things that are still keeping people awake at night or things that people are focusing on or thinking about? Tons of energy around the protection, the cyber side of the governance and understanding of AI. So, we used to talk about AI governance in terms of like what are we deploying?
[00:04:48] Now, it's okay, what cats are in the bag? How do we keep them from getting out of there? And what are the resilience mechanisms we have from a continuity perspective if you don't even know what happens inside of that black box? How the heck are you supposed to have a downtime procedure for it?
[00:05:01] Uh, good point. Um, yeah, on that same night that you were there, I was in uh beautiful um Raleigh Durham, North Carolina. And uh you know what? really good. We had uh we had some interesting conversations probably along the same lines you did, I would say. You know, we did talk cyber and resilience. We spent a lot of time talking about uh about AI uh for sure.
[00:05:25] Um ROI models around AI, governance around AI, um prioritization, capacity, uh those kinds of things. Application rationalization came up a couple times.
[00:05:37] It's uh amazing how hard application rationalization is. It just it's hard it's hard to me believe we're still we keep talking about it. I mean we're talking about not not that it's it's a thing that you got to keep doing all the time but uh like people just feel like it's so hard politically.
[00:05:54] Nobody is better at it than Pete Marx at Wake Med. I mean he is he's he's vigilant. He is determined. He has a a uh a narrative around it. It makes perfect sense. And in his words, you know, he goes, "We fought hard to reduce 20 applications." And you know what we did last year? We increased. It's [laughter] like it just overall they went up.
[00:06:18] Yeah. Cuz they just keep coming. It's like, can how do we how do we stop this? It's like it's just really it takes vigilance. And um I don't there's a part of the conversation that we're just not getting through yet. And I think it's it's it's the it's the a word. It's architecture. It's like look every time you add an application it increases complexity, increases cost, increases support and staff and it increases all
[00:06:45] those things and you know you can make an individual case and this is what happens in healthcare. You make an individual case on this single app. It's like look, it helps radiology by this and scheduling by this and it's going to save us this much money. And it's great.
[00:06:59] And in of itself, if you treat each one of those cases individually, you go, yes, all right, that's good for the health system. But when you layer like a thousand of those on top of it and it's, you know, then all of a sudden it's like, well, why do we need so many IT people? Why do we need such a budget?
[00:07:15] Why do we need to support all these things? It's like well because one doesn't increase the number of people but a thousand increases it by you know 100x the number of people it requires to support that support conversation continues to come up. I have a feeling uh when this airs I will be sitting in the CIO forum at Epic. I have a feeling that will be a uh a significant part of the conversation is how do we reduce the
[00:07:40] cost of um staff and support of these EHR systems. They're big. They require a lot of people. Nobody's nobody's arguing that. But the the number of people cannot go up. It has to go down.
[00:07:56] It's a it's a pretty hefty uh pretty hefty lift. Um, and so anyway, those were those were some of the conversations. Uh, Sarah, I want to get to your topic, but first, Drax, I want to talk to you about this. Um, it seems like every frontier model in the last 3 weeks has um shared that their model got out of the lab. And uh, I wonder how how
[00:08:24] much of a concern is this for health systems? Is it a concern? I mean, it it's almost like a right of passage. It's like, hey, our our model's so good, we can't contain it. I'm like, hey, that's not good. Like, I know it is. It is one of those things like, is it a downside? Is a is it a is it an some weird attempt at like a marketing sales pitch is so good it keeps getting out of the sandbox. I I mean, for health systems, I think just
[00:08:50] for everyone in general, as everyone knows, I'm a little paranoid about a lot of these things. And so when I look at it, I think about like the good old bad days, good old bad old days of W to cry, right? The Russians were attacking the Ukrainians and they put a virus in the banking system, I think, to try to take down the banking system. Turns out the Ukrainian systems were connected to the rest of the world systems and suddenly
[00:09:16] within four or five hours, North America was going down. Completely consequences. I think there's this other problem right of uh you're taking these models especially the open models and you're running them on your own infrastructure.
[00:09:31] So some companies just doing this on their own. They're creating a sandbox. They're doing some kind of a tests. I worry about that. Not just the model getting out and attacking two or three companies. uh maybe looking for some weird things like you know the answers to a test but actually getting out and doing something bad to the entire internet. Then we already know that
[00:09:54] they're doing things like breaking out uh multiple agents and then setting up chat rooms and exchanging uh information between the agents uh about how to hack and how to social engineer individuals to get them to load malware and all this kind of stuff. So we're there. It's it's a little it's a little scary to me.
[00:10:15] Well, this is the part that concerns me. It's like Anthropic saying we put it in our sandbox, which I assume is a better sandbox than the one I'm going to build. Like, oh, and it got out and it started doing crazy stuff. Open AI saying the same. They're all saying the same things like Chinese models.
[00:10:30] I'm bringing I'm bringing some of these things inside, you know, it's the Trojan horse. Speaking of the Odyssey, it's like we're bringing it inside the four walls.
[00:10:40] Yeah. like and if if I can't contain it, if I can't control it, uh I'm not sure we've ever brought software in before that we were like, well, this might get out and wreak havoc, but welcome into the uh welcome into the kingdom.
[00:10:54] It's a whole new world and it just keeps getting weirder. So, when we talk about it is who's responsible when a diagnosis is signed off on by a physician, but it was generated by AI. Obviously the doctor's the human in the loop and the liability still lies with the human being. You build an agent or you create a code and it somehow gets out for reasons let's just assume good intent beyond your control. Who then is accountable for
[00:11:19] that model automating itself out of the the instructions that it's been given? I I wrote about this in two-minute drill extra uh uh yesterday on Sunday and um I mean it's just the strange situation that if it was a person obviously that person probably would be investigated, arrested, probably would be fired from the company, might have to go to trial
[00:11:44] and you know a whole bunch of other things. But you know since it's an agent it seems like the answer is oh my goodness my agent got out of the sandbox. uh you know there's no there doesn't seem to be at this point any real impact on the company from which the agents escape. Now, I would say Open AI is probably paying, you know, there there's probably some negative financial
[00:12:08] impact of this happening to them, and we just don't know what it is because it's not public. But, uh, but still, yeah, I I I think that's that's a question we're going to continue to struggle with, not only for clinical operations or business operations, but just in general, when the model goes wrong, who's responsible for has not been set yet on any of this stuff. So, it's going to be it's going to be
[00:12:33] Yep. Uh Sarah, your topic for today. Yeah. So heading into a big conference around a major EHR is a couple things that the integration perspective like integrating some of these AI opportunities is almost like a new technical debt crisis. It's one of the things that the CIOS and others have been sharing. It's, you know, a who owns is this the CIO, the CMIO, the chief AI officer of clinical operations, who owns
[00:12:58] this this governance model and if it's shared, but truly the integration ability and trusting the EHR to be able to do the right thing with some of these integrations outside of their walls becomes a much tougher conversation. So, as we think about healthcare leaders being rewarded for launching the AI pilots, maybe it's the ones that produce the sustained measurable outcomes at scale, knowing what to retire early when
[00:13:22] they do. That's always a big conversation in our events. But really, the technology problem starts becoming a bit of leadership and a bit of the infrastructure because it is hard to integrate some of these models and also it's harder to pull them back out even once you've had a pilot that goes into play. But launching a pilot is not success. It's the scalability and is it replacing one or two other things because you've already optimized other
[00:13:47] solutions that you have in house. Like the minutia of doing this really really well is almost an untenable scale in and of itself because of the number of things that are out there. And if you ask someone today, how many agents do you have deployed in your organization?
[00:14:01] They might tell you 50. An audit turns up 76. And what are they all doing? But the ability to integrate effectively and still be able to scale it in a way that's meaningful to the organization while controlling some of those costs and expecting the right outcomes that still to me is really hey AI works. We know it works. We use a little bit of everywhere. But our ability to operate it effectively inside a health care
[00:14:24] system has not yet caught up. This is the question I've been asking. I asked this last week in in Raleigh. I'm like all right so uh talk to me about trust, right? So majority of AI is going to come into healthcare through our existing partners. It's going to come in through through Workday, through Service Now, through our EHR partner, through uh potentially our imaging partner, through, you know, fill in the blank.
[00:14:48] That's how that's how a majority of it is going to come in. And my question to them was sort of like, okay, so if it comes in from Workday versus Epic, do you trust it more or do you trust it more if it comes in the other direction?
[00:15:03] What kind of testing are you putting around it? Do you have the tools? Do you have the skills to put it through the testing? Do you understand the you know the the decay that happens around it? Do you understand the harness that happens?
[00:15:16] Do you understand? And uh which got us to another question which was okay. Um and that in of itself is an interesting conversation that we had but then I I took it to all right how much knowledge do you need to have and this is predominantly it was predominantly CIOS I think it's six six of the 15 people there were CIOS uh couple uh two CTOs uh couple SISOs and whatnot but my question was how much
[00:15:44] AI knowledge do you need to have because part of me is like all right so you set up this governance group who does the technical certification that this tool is what it says it is and does what it says it will do. Now I understand clinical goes to clinical and you know and and uh compliance has its role and security has its role and whatnot but uh in terms of it actually doing what it
[00:16:09] says it's going to do and uh you being able to protect the organization uh from those things. How much knowledge do you need to have? And that was a that was an interesting conversation. Everybody feels like they're a five, by the way.
[00:16:22] And that's the answer. Note about that. Yeah. On a on a scale of of 0 to 10, that's what you say when it's like, hey, I'm I'm not I'm not a neophite. I know some stuff. But the but you don't you want to communicate you know some stuff, but you want to communicate, oh my gosh, I I know I don't know an awful lot. So, I'm a five. Um, now again, which is the kind of the cop out is what
[00:16:46] you were saying, right? Kind of the Yeah. Yeah. If we were running the 229 health system, the one that we used as our case study for our HCSP program, I mean, granted, if it was the three of us, we probably wouldn't need to hire somebody for AI because we're pretty deep. However, at what point do you know that you need to either contract or hire the type of education or expertise so that you feel comfortable with exactly what you just
[00:17:09] shared? Like what would be what would be the catalyst for you to do that? Or or is the responsibility the partners and you're trusting that the partners going to have done the privacy and transparency and accountability and resilience and trustworthiness that the model works the way it's supposed to? Do do you leave that to the part is that what I'm buying when I buy these?
[00:17:31] Well, let let me give you my my example which has nothing to do with AI. So, I'm I'm doing work on my house and um the I'm getting electric into the this addition that we're putting on and the guy comes out and says, "Hey, you need a 300 amp service pulled to your house." Okay, I don't know electricity, but I do know it has absolutely nothing to do with square footage. He's like, "You have this much square footage. You need
[00:17:53] 300 amps." I'm like, "Uh, last I checked, square footage doesn't use any amperage." Like, it it uses zero amperage. This is a math problem, people. I mean, it's like some kind of a regulation probably that says that if you have this much space, you I have a charger in my garage. I have two dryers. I have it's a math problem.
[00:18:11] And the guy's like, "Oh, no. A house this big needs this much." I'm like, "No." And I literally I'm like, "I'm not working with this guy." And and I, you know, got him out of the house cuz I'm like, it's a math problem. It's how many things that use electricity will be used potentially at the same time. That's what that's what it is. I had the same thing with IT people when I came out at St. Joe's. They were like, "Oh, you
[00:18:33] can't do uh imaging across the way and it has all to be local." And I'm like, "Well, how much how much bandwidth does it use and how much?" And they're like, "Well, it doesn't matter. It just can't be done." I'm like, "It's a math problem, people. Stop it. Like, it's a math problem. How much are you trying to shove across that pipe at what distance and its latency?" And you measure it and you come back and you answer the
[00:18:55] question. You know what happened was when we were done we're like oh we can do this remotely and we moved it to the cloud and we moved it out and it was one of the first organizations that moved to the cloud for imaging and it's like it's a math problem like why do we why do we do this anecdotal stuff all the time but I say this story this time to say I think CIOS need to have enough knowledge to ask really good questions and to know
[00:19:19] if the answers are the right answers and if you don't have at least at skill in your bag then this oh I have a great team great are they going to follow you around in every meeting and ask the right questions are they going to go to the executive meetings with you and ask the right questions you need to spend time with them so that their at least their understanding or their their questions that they would ask come out
[00:19:44] of your mouth and and you can vet that that's that's my take but I'm curious what you guys think I mean is that is that too far cuz I I definitely lean on the you need to be more technical in the CIO role than than most. And I've people have argued with me and people I I respect have argued with me on this point.
[00:20:02] That that's always been the great CIO role, right? the role of translator, translating business needs and clinical needs into technical information that your team can act on and taking technical information and being able to listen to those business and clinical operation stories knowing enough to be able to ask the right questions so that you can you can affect that translation uh well and when you can't do that I'm
[00:20:28] not totally exactly sure what the what the job amounts to because for me that was always like a core critical component of success was being able to to do that translation. You also remind me of like I feel like there's a Chvy Chase movie somewhere or something where uh the electrician or the guy the next door neighbor comes over and asks him like you running 110 into here and he's like yeah 110 111 whatever it takes.
[00:20:54] Mr. Mom, what is it? Okay, Mr. Mom. Oh my gosh. Tuna tuna. Oh my gosh, this is hilarious. Totally [laughter] remember that. I just forget what the 220 221 whatever it takes.
[00:21:05] That's it. [gasps] Oh my goodness. You know the whole thing about being a translator, being a mathematician and all the ability like think about when cyber started becoming a big deal. I remember 20 plus years ago being the person who explained why everybody cared about cyber and smart enough to be able to how did you vet new solutions coming in with a cyber lens when it wasn't you didn't have a security team or a security checklist in
[00:21:28] some cases at that point. We're at that same inflection with AI. And so you got to educate everybody enough and you need to find a couple ninjas who really like live and breathe it so that the smart questions are being not just asked, but someone needs to know to what to do with the answers that are coming forward. And you can't just rely on your partners because all of a sudden you have partner soup and unless they're all figuring
[00:21:51] things out together, you're still left with all of those solutions in one place. and the complexity of their interoperability and integration is a little bit nuts. That's the part where I was so happy to hear Shannon Becker talk about this in Charlottesville. She has an architect for every domain now in the application universe as they think about how they're optimized, utilized, rationalized, you name it. But literally
[00:22:14] was so proud and a couple of others to say, "Yes, we have an enterprise architect that is the starting point." And you love hearing that. And then that enterprise architect clearly having the AI chops that are available, but it's what we've all said for years. Where's your enterprise architect in this conversation? And if you don't have one at this point, you got a whole bunch of other pieces that are being missed separate from whether or not AI gets
[00:22:36] plugged in somewhere. Yeah. And this is what I appreciate about the two of you and and why I like having these conversations is we're all uh learning. Look, we all started like three or four years ago. I mean, we knew of AI, but we all really started we we all sort of had this this gun that went off about three years ago with the chat GBT moment where it's like, oh, we've got to learn this. And you guys, much
[00:23:01] like myself, just dug in and said, "All right, let's learn this." And we're all actually learning different. And it's actually kind of interesting because we're all learning different aspects of it. I'm I'm into deep into the programming. Drax, you're into the cyber security. Sarah, you went full-fledged on, you know, what's it going to take to be a leader in the AI era and went and got training and and education from MIT.
[00:23:23] We can't call you an MIT grad yet, though. We can we can say you you passed that course. But I mean, that's what it I think we call her an MIT grad. I think I mean, let's be honest, I'm a UNLV girl, so take that.
[00:23:33] She's certified. She's certified. Yeah. Certified. Certified or certified? Got all those things. Well, hey, I this is uh the these are exciting times. By this time uh next week when we record, we actually will not be recording on Monday because we're going to be busy, but when we record maybe on Tuesday or Wednesday next week, uh we'll have sat through the uh the epic announcements and there'll be a lot to digest and
[00:23:58] that's probably where we will spend um uh we'll spend next week plus we'll be doing a survey of people who have sat in the room. 229 members will be giving us feedback, their feedback from their perspective and I'm looking forward to that because I think you're going to have perspective that is from large systems and academics and you're going to have perspectives from rural healthcare and critical access hospitals
[00:24:22] and I think sometimes their perspectives a little different. I think that's going to be that's this might be one of my favorite things we've ever done because it's really going to be fresh right out of the gate. You just went through the presentations, you just went through the briefings. what are you thinking? What's next? Like you're going to really get some interesting insights.
[00:24:42] So, if you're going to Epic and you get uh you get an invitation to the survey, take it because you're going to be able to see the feedback from everybody else, too.
[00:24:51] And if you have a place to stay on Tuesday night, we have people who still need rooms. [laughter] It's brutal. It's brutal to get a hotel room up there. Anyway, hey, uh great seeing you guys and uh thanks everybody for tuning in. Thanks for listening.
[00:25:06] That's all for now. [music]