This Week Health

Contributors

August 18: Today on TownHall Brett Oliver, Family Physician and Chief Medical Information Officer at Baptist Health speaks with Joey Bargo, Radiologist and Clinical Informaticist at Thynk Health about AI algorithms in healthcare. What are the major limitations for mass adoption of AI for organizations looking to start using it? What are the challenges in evaluating the data once implemented? What is an organization's obligation to transparency when using AI?

Webinar: Don’t Pay The Ransom - Thu Aug 18 @ 1pm ET / 10am PT

Is your health system paying out for attacks on your data? How can we stop the cycle? Rubrik is offering incredible insight into the arising issues in cyber resilience in ransomware attacks, cloud data management with securing Epic in Azure, and unstructured data. In this webinar, we analyze the best practices to initiate in our hospital systems. Register here

Transcript

This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

Today on This Week Health.

in medicine specifically, data sets are very, very small label.

Datasets for clinical imaging are microscopic You really have to be able to evaluate the size of the data set the organizations where these data sets were obtained and even beyond that, what scanners were they performed on?

Welcome to This Week Health Community. This is TownHall a show hosted by leaders on the front lines with interviews of people making things happen in healthcare with technology. My name is Bill Russell, the creator of This Week Health, a set of channels designed to amplify great thinking to propel healthcare forward. We want to thank our show sponsors Olive, Rubrik, Trellix, Medigate and F5 in partnership with Sirius Healthcare for investing in our mission to develop the next generation of health leaders. Now onto our show.

Hello Brett Oliver here, family, physician CMIO for Baptist health in Kentucky and Indiana. And I am excited today to have Dr. Joey Bargo with me. He is a practicing radiologist, but for today's discussion, I'm gonna focus a little bit more on as other job as the board certified clinical informaticists and the vice president of clinical information at think health Joey.

Welcome. It's great to see you and have the opportunity to pick your brain a little.

Thanks, Brad, happy to be here. And I appreciate the invite to uh, come on and have this discussion today.

Absolutely. Well, could you start a little bit just with your background? Like how did you get into AI machine learning and then maybe as a correlator to that, how does your physician education and influence, you know, the companies you work for?

Just how you approach things with AI.

Sure. Yeah. So as you mentioned my background in formal training in medicine is within diagnostic radiology. I still practice clinically about 50% of the time did residency training at university of Kentucky. And even as early as in residency really started develop an interest.

data structures in general and more specifically unstructured data structure, like we see in medicine and data mining specifically out of radiology reports for about the last 10 years sort of incorporated in with clinical practice, I've been heavily involved in. Not only my own organization with clinical informatics population help type projects but also involved with ACR data science Institute RSNA in a lot of the folks that are leading these initiatives in machine learning and artificial intelligence.

Just in terms of my role with industry and the way I see, physicians and providers roles in industry in general. Sometimes it's got a little bit of that icky feel amongst clinicians, but I think, just like hospital administration, I think those partnerships with those industry government or how physicians are really able to guide.

And make healthcare, what it is. All those partners really have to come together. And it requires physician leadership to really massage that into what we wanna see it become which is ultimately good for the patient.

Absolutely. I'm curious when you were, starting to play around with the unstructured data in your residency, were there clinical problems that you were seeing that, that were going on unaddressed and that's kind of what got you into it?

Or was it more just the technical aspect of saying, Hey, we could do something with all this unstructured data?

Well, certainly the technical interest is always been something that, I've had and drives me, but it was, definitely a clinical issue that still even exists today, which is that of.

data mining specifically around radiology reporting, radiologists have, have been leading the way in having our electronic notes be in the system for a very long time. But even with all the structure we can apply to the reports and all the different things we can do. The data mining has been a significant limitation.

So early in residency, we started on a major project at UK to start structured reporting, really one of the first in the nation to create a report repository with the end goal of data money.

Gotcha. Okay. Well, what, true AI algorithms are actually being used in radiology practice today?

Yeah, I think that's a, great question and important to look at, because I think not unlike, maybe Netflix 10 years ago they were starting to use AI and a lot of folks really didn't even know. It, it was in use in a product they were using. daily I think similar is true within radiology realm right now, radiologists, including myself and probably a vast majority of radiology groups across the nation are using AI products every day, maybe even unknown to other providers hospital leadership and patients.

Some of the use cases that we use daily more often revolve around stroke imaging at this time is probably the most common use case. In that things like if there's an obstruction in a blood vessel or an occlusion in the intercranial vasculature, we have AI algorithms that can help detect that and then give us really a second set of eyes.

The other types of things we see it used for pretty commonly are,b are detection of intercranial, hemorrhages that are acute. Again, it's that second set of eyes sort of helping us, but it's also helping to do things. like Drive that study to the top of the list in terms of priority. So we're, really seeing AI become sort of an assistant second set of eyes.

It's not really gonna be taking my job anytime soon. I can attest to that, but it is very beneficial in, having that sort of backup. Additionally we see it used in what would be considered tad type products for both mammography also for things like pulmonary nodule detection That's a pretty common use case that we're using those for some of the, you know, lesser known things would be things such that I'm working on with Thynk health would be in the use of machine learning, artificial intelligence.

After the report has been created to pull out incidental findings and those types of things to help make sure those patients don't slip through the cracks and they're appropriately managed

Gotcha. Gotcha. It's good stuff what are the limitations to the approaches that companies are taking with AI today?

And are there solutions to overcome those or are there always gonna be limitations that we see?

Yes. And yes. Really the, I see some of the major limitations right now that will prevent what I would consider mass adoption at scale. Are a couple first of all is really on the vendor and what I consider the usability of the software that there's still somewhat disparate in that, I may have a software AI software looking for large vessel occlusion.

I may have one looking for bone fractures. I may have one looking for pulmonary nodules. These are still fairly disparate systems. and anytime you're having to break out of your traditional workflow, that's not well incorporated. There's room for error there's room for oversight. And there's just, obviously inefficiency.

That's a major problem. I see. But along with that problem, probably more, maybe even more important is the fact that the user interfaces still aren't really designed to capture end user feedback in a way that can. help Maintain quality assurance. There's really no feedback mechanism. That's convenient that I can click on something that's a false positive.

And that gets reported back to the vendor. Not only is that a limitation in the ability for them to constantly improve their algorithms. It's probably even more important that we don't understand at a institution level, for example, that what our true sensitivity specificities. are Because we're typically using their generalizable globalized algorithms which may or may not be performing well at our organization.

And we really don't have that information to understand what that performance level is.

Does that make the initial assessment for an organization of what data set the algorithm was trained on that much more important?

I think. Yes. So there, are two levels to that, you know, I think what we are seeing at some organizations and what we should see at most, anyone that are in taking on the responsibility of employing these algorithms are not only a machine learning team who really understands those initial data sets and helps.

evaluate Not only how the algorithm works, but also how it may work at our own organization and testing, but setting up a continuous QA process to maintain that over time, that algorithm is getting better and potentially not causing harm to patients or, or certain inefficiencies in workflow.

That makes sense. So that's one challenge I'd really like to understand, cuz you've got to work with a lot of different organizations. What are some of the biggest challenges you see, besides one we just mentioned when folks are evaluating algorithms, AI offerings, mm-hmm .

Yeah. So one of the fundamental issues that we have to address, and, I have some theoretical ideas on how that will evolve in the future, but one of the theoretical issues that causes limitation of these algorithms being generalizable across organizations, Is the in medicine specifically, data sets are very, very small label.

Datasets for clinical imaging are microscopic in terms of, a Google algorithm that can scrape a picture of a dog right off the web. There's a lot of pictures of dogs out there that are labeled so there's certain pathology or pathologic findings on imaging. That there are just no representative cases for and you really have to be able to evaluate the size of the data set the organizations where these data sets were obtained and even beyond that, what scanners were they performed on?

What technical parameters now, AI algorithms typically have mechanisms to work around those variabilities. But it's certainly something that really has to be evaluated pretty closely to anticipate what the generalization what the performance is gonna be at your own site. πŸ“

πŸ“ We'll get back to our show in just a bit. I'd love to have you join us next Thursday for our webinar, don't pay the ransom. Cyber threats are mounting everywhere, especially in healthcare leaders. Thomas Jefferson university health, as well as St. Luke's university health system and rubric are gonna join us to discuss solutions around protecting all healthcare data, even epic in operations on Azure. This webinar will be on Thursday, August 18th at 1:00 PM. Eastern time. You can register now at this week, health.com or by clicking on the registration link in the description below. Now back to our. πŸ“ πŸ“

Do you see these organizations utilizing? Cause I think one of the hesitations from clinicians is Where the double blinded placebo controlled studies that this is better and, medicine and the information that you and I are supposed to is moving so rapidly. I'm not sure that's We're gonna be able to go that route with everything. And do you see some just running these algorithms sort of, behind the scenes with real data, but perhaps not showing it to a clinician until that they can put three or four months of data together and compare to the standard of care, utilizing that method or has that got holes in and of itself as well?

Yeah. So, I, I think that's the responsible pathway is running an algorithm behind the scenes without really displaying that. And I think that's a pretty common mechanism used by folks like epic, for example. We've seen that in use in some of their machine learning algorithms to evaluate that performance over a time period and be okay with saying, Hey, this is not gonna work.

We can't refine this model well enough because of something in our data that we don't use it, or we have to be okay with turning off models, if they're not performing. And that comes in with the continuous process. I think another approach that we will probably see in the future as, technology has sort of allowed for these transitions with more complex recurrent neural networks and convolutional networks and transformer network.

More accessible to companies is something we're actually working on as well with a large university and been championed out of MIT AI labs. Is a concept of unsupervised machine learning for labeling of data sets and constant reinforcement. Of the algorithm from end user feedback that's occurring anyway.

To sort of break that down. It's almost a multimodal form of learning. So for example an image comes across with a, a lung nodule. That's three centimeters. The AI algorithm detects that, or it doesn't detect that. And then using technologies like we would utilize at think health in the way of processing that radiology report information.

You can bridge those two learning mechanisms together to see whether or not. The radiologist interpreted that as a pulmonary nodule, whether they saw it in the AI algorithm or not, doesn't matter. It's whether or not they agree with that AI algorithm put it into their radiology report. And then we mesh up those to evaluate performance of the model.

And I believe. That will be somewhat of the golden grill in these universal lesion, tagging systems. You can just get so many more use cases so much more rapidly, but also that's gonna be the important aspect of our QA going forward. We're not gonna burden the clinician. To say whether or not the algorithm was right.

We're gonna take their end work product, see how they use that and see how that correlates into what the AI algorithm originally proposed. So, I'm really excited to see what that develops into. , I think that's, probably the most scalable path forward in medicine right now to achieve.

Larger label data sets and QA

makes sense. It's exciting. It really is. So you mentioned something earlier, you least alluded to it. I wanted to switch gears a bit and talk about transparency of these and what do you think what's the obligation for an organization of practice a company when we're talking about transparency regarding the use of the algorithm in practice, Is there a certain level of seriousness of an algorithm that needs to, you know, if we're talking about sepsis, then a provider needs to understand that this information being presented is being presented by an algorithm.

Does the patient need to know? And at what level that's the one thing that I've struggled with a bit something very simple, like an appointment opening. Does the patient need to understand that that was created by, an AI algorithm versus maybe something on the radiology report?

For some things we're talking about. Thoughts on transparency.

I think what we're currently seeing is it's almost like for example, a, an airplane pilot who 99% of the time is probably using some form of assisted flying of the airplane. Right. It's that other few percent of the time you've still gotta have a pilot for and we all want 'em on there.

Right? I think it's gonna be the same. With the clinical decision makers. I think first of all, , it's required that once an algorithm delivers information we as a provider take that information and review where it's coming from, and then we make the next step. That way we're always responsible.

And I think that's the way everyone will want it to be. And we're depending on the algorithms to make it easier for us with that said, I think. It's absolutely should be required of the vendor to present that information in a way that makes it easy for them to quickly review, understand how it came about its decision and understand the confidence interval of, are, are we 80% sure on this?

Are we 90% and give a really nice breakdown of what the identified parameters. That's helping to make that decision. I think that takes a lot of the black boxiness out of these. And when you really come down to it at the end of the day, there's not as much black boxes type effect, as we would think in algorithms, in medicine, because they are fairly variable and parameter driven.

So there's always, sort of that explainability. It gets a little more complex with computer. Especially unsupervised approaches. You can have inserted bias into these algorithms. That's gonna deliver some type of output. That's just not expected and just not correct.

And that's a little hard to explain to the end user. But it, again, it's on the vendor to present that in a way where they understand how that decision's being made fundamentally and can quickly either agree with that or disagree with it to drive management.

So you think there's less proprietary information in the algorithm itself per se, that there's enough.

That can be exposed, that a clinician can understand kind of where it came from. Cause that. That's a tough thing to understand here. This, AI algorithm is telling me X is the best course of treatment, and yet I'm not allowed to understand how you got to that answer. You know what I mean? But you think you're saying that for a lot of the products that are out there, you should be able to reveal some of that without showing your secrets off.

Yeah, I believe in medicine, it's a requirement. I think the algorithms should be reproducible and they should be able to be tested and reproduced. And certainly with as much open data as a company can provide. But I do believe that fundamentally, there's still things that are fairly hidden, right?

I mean, so for example, if we present to a clinician that. 10 parameters that went into a sepsis decision. You can show the relative waiting of those decisions without demonstrating the whole convolution neural network that may have been involved in recalculating those percentages. Okay. And it still, still relatively makes sense.

Yeah, I gotcha. Well, let's wrap up with a question of where do you see the biggest opportunities right now in healthcare, obviously, radiology. I mean, I would think there, those are some opportunities right now for organizations, for sure. To please comment there, but are there other specialty areas that you see rapidly developing where they might be a couple years from now?

Sure. Yeah. I, I have a lot of involvement with different specialties in the industry. I think radi. I think it's a big one. Of course, but there are probably bigger and even more impactful opportunities within, clinical, more what I would consider, patient interacting type specialties in general, in that we have the opportunity with the amount of data that we have now flowing into large systems.

Like. To use AI to become really an assistant to facilitate very efficient workflows, make sure things aren't slipping through the cracks and delivery of. Information to the clinician at the right time, the right information, the right time to really, really impact on how efficiently they can move throughout the course of the day without burning them out with a thousand different clicks or having to read , all these different, places in the chart.

But also in a positive way to the patient to make sure certain things aren't slipping through. I think artificial intelligence in the use in the exam room and note assistance I think is, gonna be huge. I. As clinicians, we're doing a lot of duplicative work that in other industries, you're just not seeing that because they've developed mechanisms to translate the work you're doing at that time into documentation.

I'll be excited to see where that goes. I know a number of organizations are experimenting with that as well as big vendors I think that'll be a big one. Other places that we probably don't think about as much, unless we're in those niche specialties, but in things like surgery, And in navigational, interventional pulmonology the robotics units, for example really employ a lot of machine learning and AI in evaluation of how to get to certain lesions, how to do certain things, but also evaluation of performance and experience level of the surgeon.

Which is constantly re helping them train in different mechanisms to improve their performance.

Oh, wow. I wasn't familiar with that. That's awesome. That's some pretty cool stuff. Yeah, it, it made me smile as some of the things in the exam room because. We're not supposed to text and drive.

Right. Cause you don't wanna be distracted. And right. Usually whoever's leading the meeting is not taking notes. But yet as a clinician, I'm supposed to listen to the patient, gather that in, start making some plans and some ideas. But at the same time, let me document my note. Right. And it just seems like a crazy system that we've developed, but yeah, I'll be excited to see where that goes.

Joey. Gosh, I feel like I'm just getting started here, but I appreciate your time and hopefully we can connect again, cuz I've gotten lots of other questions , and your wealth of information. So thanks for joining us.

I really appreciate Brett. Thank you

I really love this show. I love hearing from people on the front lines. I love hearing from these leaders and we want to thank our hosts who continue to support the community by developing this great content. We also want to thank our show sponsors, olive rubric trellis. Mitigate and F five in partnership with serious healthcare for investing in our mission to develop the next generation of health leaders.

If you wanna support the show, let someone know about our shows. They all start with this week health and you can find them wherever you listen to podcasts. There's keynote town hall and newsroom. Check them out today. And thanks for listening. That's all for now.

Want to tune in on your favorite listening platform? Don't forget to subscribe!

Thank You to Our Show Sponsors

New Shows for 2022

Insights - This Week Health
This Week Health Town Hall
Keynote - This Week HealthSolution Showcase This Week Health
Newsday - This Week HealthToday in Health IT - This Week Health

Related Content

1 2 3 158
Amplify great thinking to propel healthcare forward and raise up the next generation of health leaders.

Β© Copyright 2022 Health Lyrics All rights reserved