Welcome to This Week Health Conference. My name is Bill Russell. I'm a former CIO for a 16 hospital system and creator of This Week Health, a set of channels and events dedicated to leveraging the power of community to propel healthcare forward. Today we have an interview in action from the Fall Conferences on the West Coast.

Here we go.

 All right, this is part of our Interviews in Action series, and today we're joined by Paul Milligan, Director of Product Strategy for IQVIA. Paul, welcome to the discussion. Looking forward to it.

Yes. Thank you. Thank you for inviting me. I think this is a great subject.

you think anyone's interested in this subject right now?

Well, there's a lot of us over here who are really interested in it. So, yeah, absolutely really keen to kind of share our learnings and the impact that it's having across all of

healthcare.

All right. So we're going to talk large language models and we're do a dump over the next you know, 15 minutes or so.

But before we get there, talk about IQVIA. What is it that, you guys do and what do you do with IQVIA?

So we are part of the NLP team within IQVIA. So all of my products are NLP products, natural language processing. Some call it the original AI technology, it kind of replicates how people write information down.

And so we extract that information out of, any kind of text. Obviously that can work with patient records, it can work with scientific literature and so there's a whole range of things that you can do by extracting information out of those sources to kind of find new information or to answer specific questions or put the information into a kind of a key process that will allow you to enrich the data that you've already got.

Now we used NLP like it was the magic pixie dust. Oh my gosh, it could take All this stuff in healthcare and move it in there. And now we're talking about LLMs, like the way we used to talk about NLP. What's the distinction? I mean, they're different, but what's the distinction?

Yeah, that's right. So NLP is the, I guess, the general approach to understanding how people write things.

And so it's very much Text based either you're reading text or you're generating text with, these new large language models, as opposed to large language models themselves, which is kind of a technology that allows you to do, it allows you to do NLP, but we've also seen it allows you to with DALI, it allows you to generate images as well.

It allows you to create, avatars and movies. And so it's not just, text itself. It's going beyond that as a kind of a fundamental technology.

Now, we've grown to trust NLP over the years. It doesn't give us any hallucinations. It doesn't give us any new information. It literally looks at what is there and it's able to extract the meaning and the value from that.

So, how are we going to get there with LLMs? Or do we even need to get there with LLMs? Are they just different?

A lot of it is how complementary they are to existing technologies, particularly in a regulated space. Because, the regulations are there for a reason. They're to protect the data, they're to protect the subjects, they're to protect the, the patients.

And so we see LLMs as just another tool in that toolbox to extract information. Now, a lot of the LLMs and, we, we often talk about the, you know, we talk about things like GPT, but we also talk about them in general terms as generative AIs, because they are more about generating new information or summarizing information from a set of data, as opposed to doing the information extraction part of it, which as you said, is, has been going on for, for many, many years in this domain.

So we see it as almost. You've got the two sides of that equation of understanding and then generating natural language. So it, maybe won't change very much that extraction phase, but could open up new ways of allowing physicians to, for example, interact. with the data, it will allow you to ask a question of the data in a natural way that will then be able to be interpreted and, give you a natural answer in response, as opposed to a table of data or, a chart or something like that.

It's amazing in healthcare how, in how many places we're doing summaries, right? So there's, there's human beings going out there and doing summaries. If you have a complex case in the ICU and whatnot, you're generally pulling a lot of different records and creating summaries. A lot of times staff, that's nursing staff or somebody else doing that, creating those things.

On the billing and revenue cycle side, you have summaries. You have all this information coming in and you're trying to create a summary in a way that people can understand it. I mean, that's the number one thing that we hear is that they don't understand it. And I mean, that's the tip of the iceberg when I interview CIOs, they see these large language models as a real opportunity to fill that.

I mean, there's a lot of use cases they're looking at, but that's the one that they're looking at going. Hey, we've been testing this a little bit in this area. It feels really powerful in summarizing things and we do it all over the place in healthcare. Is that what you're seeing as well?

Document summarization really, so far, as you said, the sweet spot how we've seen it being used.

We've also seen it being used for kind of question answering, so more like a dialogue system. But, yeah, if you can.. get a large amount of complex data distilled into something that's much more, simple to consume, easy to action , then obviously that's a really good thing.

And, and all you need to be sure about is, is the information in the summary actually representative of the bigger data set, right? And that's, that's then the challenge.

Yeah, so how do we guarantee that accuracy? I've heard this layering of LLMs kind of thing. I've heard a lot of different kinds of approaches.

I'm just wondering, is there one approach over another that's going to, I mean, guarantee is kind of a strong word, but it is healthcare. We sort of need to guarantee the accuracy of the data.

Absolutely. And You almost need to guarantee the process as well as the data. That's the kind of the compliance angle here.

The way that we see that working is, it's almost like you've got multiple AI technologies, as you said, and you can either think about them as layering on top of each other, or you can think about them as almost adversarial. So one is doing something and another one is trying to prove or disprove what was done by, the first one.

And that way you can almost kind of capture the information in the big set, capture the information in the little set, and then compare them and see if, oh, actually you were mentioning the same chronic conditions in the summary as you were in the, kind of the sentiment or the relationships that are expressed in the summary.

are reflected in, the larger data set as well. So there's a, there's a large amount of kind of checks and balances that we, want to put in at the moment on top of those LLMs because we, yeah, we, we cannot quantify simply the, amount of, hallucination that, any individual model is, creating in any particular task. 📍

  📍 We'll get back to our show in just a minute. We have an excellent webinar coming up for you in November. We had an excellent conversation about AI in September with three academic medical centers around the topic of artificial intelligence.

It really was exceptional, and we released it on our podcast channel so that we could share it with a wider audience. I wanted to explore that topic a little bit more, and I asked a couple of additional health systems to join us to explore the use of generative AI and other forms of artificial intelligence to see if we can identify some pragmatic approaches to how health systems are looking at taking advantage of this technology.

The webinar is on November 2nd, 1pm Eastern Time, 10am Pacific Time. You can reserve your spot on ThisWeekHealth. com and one of the things we love is that you can submit your questions in advance and we can make sure that we, answer those questions and keep the webinar relevant to the things that you're looking to talk about.

So, please join us November 2nd, 1 p. m. Eastern Time, 10 a. m. Pacific Time. Now, back to our show. 📍  

So you're almost creating a, like a quality assurance LLM that is antagonistic and saying it's looking at what was produced by one LLM and it's looking at the original data and it's going. Hey, I, you know, I don't see that. Let's just say something like this. This interview will be turned into a transcript.

It will actually go through ChatGPT and it'll come out and it'll say, Hey, here's all the things that Paul said. And it'll have five quotes. But in this model, the ChatGPT will come out and say, Hey, here are the five quotes. And the QA would go, Hey, I don't see where Paul said this in the original text. Can you verify that that was actually said?

And there, you create this sort of back and forth until we get. A higher quality item is that, I mean, that's, that's an interesting model.

only way to really guarantee it and, and obviously as I said, it's really hard to guarantee it. I think the other point to make is that the adversarial part doesn't need to be another LLM.

It could be a different NLP technology because you almost want it to be a different kind of technology to stop it kind of. They both might have

the same bias

or problems. Copying its own homework kind of thing, where there's been systems in the past that have kind of encoded information and then decoded it in such a way that it was kind of hiding a lot of the information in the summarized version.

And then kind of expanding it out again. So we've seen that in the past. And the more that you can kind of have complementary technologies, the better your confidence that, the system is, behaving as, expected.

So, I mean, we're talking about healthcare here. Is IQVIA focused predominantly on healthcare?

Absolutely. Yeah, it's a combination of kind of medical records, but also there is a large amount of kind of prescription data and structured information around kind of the commercial side kind of drug delivery and development. The, the two things kind of intersect at the level of obviously prescribing medications to patients.

And then the, the clinical studies you know, all of the, the real world evidence around those clinical studies as well as the, patient records.

So what's the product? What's the product that a health system would be familiar with?

So an example of a product that we make is We make a search engine type product that allows you to point out a set of records. It could be hundreds of thousands, tens of millions of patient records. And you can combine these search terms in the way that you can, do a Google search. But you can combine it with... A list of diseases, a list of indications, a list of adverse events, to build up a really good pattern of information from that, list of patients.

It allows you to, for example, select cohort of patients who have particular inclusion criteria. You could exclude people as well from that list. And it basically turns that unstructured kind of low value information into something that's really structured and gives you kind of really clear information.

And, and one example of one of the things that you can do there is, for example, if you're trying to kind of identify social determinants of health in the free text, in a, in a patient narrative, because a lot of that information is not coded. It's not in the structured information, but you can ask these questions or you can.

Apply this almost like a filter to say which of these patients, for example, has difficulty ambulate, uh, ambulating doesn't have English as a first language and really allows you to find , those little cohorts of patients in a way that you, you can't really do it without kind of inspecting the, the underlying text.

Yeah. I mean, that's, that's, that's really interesting. The SDOH. the use case there is, is really fascinating because that, at this point, we aren't coding that information. It does end up in the unstructured note, and more and more, it's becoming critical, especially as we up moving down this value based care models, starting to take on risk and those kinds of things.

It's understanding. the things because we know that the actual health care is only 20 percent of the outcome. So, those things need to be considered. As we look at health care, obviously we have the quadruple aim. One of the things is the cost of care.

The cost of care in the United States will be a hot topic over the next Whatever, until November of next year it will continue to come up. I guess my question is, do you see these AI models? And I don't, I, I'm not sure people, I, I looked at the Gartner Hype Cycle recently, and they had all the different AI technologies laid out on this thing.

And I think there was like like 70 of them on this. And they're all at different points of the, Hype Cycle. And some of them are actually emerging. They're very... Like NLP would be one of the ones that is, emerging, good use cases, trusted. It's already gone through the trough of disillusionment.

This is tried and true. The LLMs, though, are at the peak of the hype cycle. We don't, I think we're going to find out over the next year or so. Where it falls down and those kinds of things will hit the trough and whatnot. But there's just a ton of these. I think the hope is that it brings two things.

One is the clinician experience and the cost of care. That we can drive efficiencies into how we deliver care and we can make the administrative burden of being a clinician better. How much of an impact do you think it's going to have on those two things?

I think the way that we can make the process more efficient is obviously a key driver.

I think that might require different... So, chat GPT is, the one that everyone thinks of. But as a cloud based system, it's just not going to be kind of cost effective. So it's going to involve probably fine tuning, different LLMs, things that are in house, things that are not on the cloud as a way to kind of.

get the value of those LLMs to, be able to, for example, to summarize the information, but at a cost point works for, tens of millions of patients. So I think that's probably going to be quite a key consideration, but in terms of Unlocking kind of value from a lot of the information that is already in these patient records.

We already see that with our own NLP tools and by, engineering these large language models by fine tuning them on your own data. We can see that, that, that there will be kind of performance and accuracy improvements and as long as we, kind of capture and make sure that the hallucination is minimized as much as possible by using those kind of QA systems that I spoke about earlier, then absolutely I can see how we can improve the value and also increase the efficiency of these processes.

Paul, I want to thank you for your time. This is an exciting time to be in this space at the intersection of technology and health care. I think. We've digitized the medical record and that was a long slog for us, but now we're starting to see the possibilities are on the other side of that now that we have these technologies and computing power and everything and, an exciting time.

I love, I love the work that you guys are doing. So thank you. Thank you again for your

time.

Thank you, Bill. And thanks for the opportunity. Cheers.

Another great interview. I want to thank everybody who spent time with us at the conference. I love hearing from people on the front lines. It is phenomenal that you shared your wisdom and experience with the community and we greatly appreciate it. We also want to thank our channel sponsors who are investing in our mission to develop the next generation of health leaders.

They are CDW, Rubrik, Sectra, and Trellix. Thanks for listening. That's all for now.