This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

Welcome to This Week Health Conference. My name is Bill Russell. I'm a former CIO for a 16 hospital system and creator of This Week Health, a set of channels and events dedicated to leveraging the power of community to propel healthcare forward. Today we have an interview in action from the Fall Conferences on the West Coast.

Here we go.

Alright, here we are from the Health Conference. Another interview in action. We have caught up with Yohany Salad with UC Davis Health System. CMI, CMI, you have so many titles on there. Tell us what you do at UC Davis. Thank

you. I'm a CMI of Digital Health and Debut Innovation for UC

Davis.

Innovation? This is the right conference to be at then for innovation? It is, absolutely. So what kind of things are you seeing here? What are you Is there a certain problem you're trying to solve for UC Davis? So,

a lot of the problems we're looking at at UC Davis are applicable to the problem that we're trying to solve as a healthcare engineer.

Specifically, how we become more transparent. How we can become more operational efficient. How we actually build faster and evaluate a lot of our clients. So, what I'm looking for from a UC Davis perspective, looking at the vendors, is we are finding great partners, especially on the early stage, that can come and look at some of our operational or clinical challenges and we can partner with into delivering more scalable solutions.

Specific. We've talked to clinician burnout. We've talked about operational efficiency. Are there specific areas that you're focusing in on right now? So,

a lot of healthcare systems have a very similar spectrum of the problems. If you look at that, we do not have access to workforce, right? We have to reuse and hire a lot of the specialists, especially across the nursing specialty and the clinical.

So we're looking on everything that's helped us to... FHIR, FASTER, unbordering FASTERS, as well as increased retention. So from a clinical effectiveness, we know that our workforce, and speaking as a practicing clinician, just cannot deal with all the administrative burden as we see for a long time. So what we're looking at is what kind of a solution we can bring a point of care to help them manage that.

That's range from artificial intelligence, we're looking at from the ambient technology and better administrative tasks, visibility, recording and documentation, to something that just helps you to get additional 10 15 minutes per day, but already allow to relocate this time for your self care and spending more time with your friends and family.

Every booth talks about AI. Absolutely. Now there will be some aspect of AI washing, but there's, to a certain extent, it's AI has been democratized. Like, you and I can probably pull it up on our phone and interact with some sort of generative AI solution. It seems like the tools are getting more accessible.

These partners are starting to integrate them in all different ways.

How do you bring those into the health system? How do you make sure that the algorithms are going to serve your community, serve you, serve... I mean, how do you make sure there's trust built around those things? That's a great question.

I think from our perspective, A, whatever you bring AI or non AI solution, you need to be very clear what problem you're trying to solve and what's your KPI swooping, right?

So, AI is one way to build a solution, and junior to AI, it's... It's another way, right, to build a base on AI, but then the question is, what does it give you, right, and how do you evaluate success and effectiveness in the long run? The second thing is, how do you even test non deterministic AI systems? I think that's something we are...

We are actively working and partnering with healthcare systems as well as the federal regulator. As you know, for example, UC Davis launched Valid AI initiatives yesterday with 30 healthcare systems coming together and we'll be discussing what's the best way to look. At the same time, when you look on the CHI, when you look at FDA regulation, They talk about assurance swap and coming up with a set of principles as we may use for classic AI and less deterministic AI.

So I think there is a lot of unknown. And what's making it even more complicated, that's, as you mentioned, building a... MVP that's capturing your imagination is very easy and the challenge with a MVP

versus building an

enterprise level production system. And I think a lot of that's where the current devil in this particular detail is. Because we still don't really know what's happened with a lot of our large language models. At scale, there's a time. And we need to exert a lot of the careful and thoughtful approach before we actually scale. 📍

  📍 We'll get back to our show in just a minute. Having a child with cancer is one of the most painful and difficult situations a family can face. In 2023, to celebrate five years of This Week Health, we have partnered with Alex's Lemonade Stand all year long with a goal of raising 50, 000 from our community.

We've already achieved that goal and we've exceeded that goal by 5, 000, so we're up over 55, 000 for the year. We want to blow through that number. We ask you to join us. Hit our website in the top right hand column. You're going to see a logo for the lemonade stand. Go ahead and click on that to give today.

We believe in the generosity of our community and we thank you in advance. Now back to our show. 📍

 You talk about deterministic. In deterministic models, we can... We can be transparent with it fairly. Probabilistic models, though, have, we talk about hallucinations. It's almost been over talked about. That's the nature of probabilistic models, like Chad GPT, which is the most common, which is why I bring it up.

But essentially it's, if you ask it the same question ten times, you're likely going to get seven answers that look very similar. And three answers that are going to be in a different direction. And maybe 70 30 is not

great, but... So, a lot of the sequence to sequence models, right, they're predicting the next step.

You're right, we'll give you a different language. When we look at that from a clinical perspective, right, I'm not as concerned about the language itself as long as the content inside this. remain consistent in truth, right? And I think this part is critical because if I'm still communicating right information to you using a different word selection, not a very conservative, right?

As long as appropriate words. If my reasoning, right, if the things I communicate to you is actually non consistent and varied, then the question is, how do you properly... Monitor, right? What kind of information being delivered and what are use cases can be used this model for?

The large language models are very good at summarizing.

Allegedly. Allegedly. Well, yes, we still have to do people have to remember. I mean, we're like a year into this mass adoption of this, like, you know, Chats and PD goes through a hundred million users as quick as... Anything we've ever seen, and so everyone's playing around with these things.

Including physicians, including others. And the feedback we're hearing, again, not peer reviewed kind of studies and those kind of things, is you can take a lot of things. It does it does summarize those things very well. One of the use cases I've been talking to people about is when you get an ICU case, generally you have to take a lot of information.

Potentially from a lot of different health systems. Potentially a lot of unstructured data. So now we're talking about a lot of layered technologies. You're using OCR, you're using NLP, You could be using RPA, you could be using a large language model. What I'm hearing is, that use case is very labor intensive.

To pull all that stuff together and make those summaries. Anyone who's used the large language models is like, Look, I can give it, like, 100 page PDF and say, Give me a one paragraph summary, and it comes back very quickly. But in healthcare, where does that fall down? Where does that kind of model fall down, and

how do we make it more accurate for ICU care? I mean, that's pretty... It's a great

question. I think consistency name of the game, right? You need to ensure that the information you extract from your non structured data... is truthful and consistent, right? You need to ensure that information that's being grouped, analyzed, summarized from those extracted concepts, right, is also correct.

So, you cannot afford information loss, right? And I think what's concerned me from informatics perspective, even though I'm extremely passionate and bullish about the premises of a generative AI in the medicine, We need to ensure there is not a lot of noise and destruction on every step of the information transition, right?

How are we going to verify all this? How we can ensure that all the content that's been destructed from non structured data, as you mentioned, are the, A, all the content we're looking for, B, in the form and the view that we're looking for, right? We, we currently don't really have a ground truth to that, right?

Similar to summarization. A lot if my care and my critical care depends on the insight and the summary from hundreds pages, right? How you can ensure that's the right insight around the right concept. Because can it generate me a summary? Yes. Can it generate me the optimal summary that will drive my proper clinical decision on this point of care?

I think that's the

most important question. Is there a technology you've seen on this floor? I saw a security one that really piqued my interest. I'm doing a little research on it. Have you seen, I mean, there's so many booths here. I'm sure you didn't stop at all. It's amazing.

So, I see a lot of the junior FAI tool, right, as a feature.

Like, some people will work on the infrastructure there. Maybe a large place, maybe somewhere smaller. At this particular point, I expect almost every product you have to certainly be great at AI or traditional AI. Because right now, that's just the way you build efficient software. Do I see any company that's right now creating a full spectrum of a tool that I'm really wanting to deploy right now?

Not yet. But there's a lot of great discussion and a lot of them are coming on the level of a healthcare innovation, federal regulator. Right, and the initiative. So I'm very optimistic that's in 12 to 24 months We actually will start seeing a lot of the platforms and actually helping us with this safety monitoring and trust

Thank you

Another great interview. I want to thank everybody who spent time with us at the conference. I love hearing from people on the front lines. It is phenomenal that you shared your wisdom and experience with the community and we greatly appreciate it. We also want to thank our channel sponsors who are investing in our mission to develop the next generation of health leaders.

They are CDW, Rubrik, Sectra, and Trellix. Thanks for listening. That's all for now.