Today in health, it we're going to revisit the MD Anderson Watson. Project. And we're going to look at it for clues as to what we should avoid. As we head back into the waters of artificial intelligence and maybe some things around some costs and projects and practices and those kinds of things. We'll take a look it's Friday. My day to just ramble a little bit about these topics.

And that's what I intend to do. This is an interesting one. My name is bill Russell. I'm a former CIO for a 16 hospital system. And create, or this week health set of channels and events dedicated to transform healthcare. One connection at a time today's show is brought to you by Panda health. Digital health is hard and Panda makes it easier. Quickly and comprehensively. That digital health solutions and be fully prepared and informed for your next meeting. Panda helps health system leaders make confident decisions about digital health without the complexity and burden of figuring things out on their own.

They help you to get smarter faster through peer input, marketing intelligence and advisory services. Check them out at this week. health.com/panda. Hey, this new story will actually be on the new site. So check it out this week. health.com/news. To make you aware of the fact that we did exceed $150,000. For Alex's lemonade stand and cures for childhood cancer. You are a very generous community.

We want to thank you for being a part of that. And if you are, haven't been a part of it yet and would like to be, go ahead and click on our website. And in the top right-hand corner, you're going to see the Alex's logo. Go ahead and click on that to give today. Finally we hope that you are already mentoring somebody and listening to these podcasts with them.

If not share it with a friend. Share it with a colleague you said is foundation for daily or weekly discussions. On topics that are relevant to you in the industry. They can subscribe wherever you listen to podcasts. All right. This one will be an interesting one. And so again, this is a posted up on our. Website, but big data bust MD Anderson Watson project dies.

And this is from February 22nd, 2017. All right. So this is a cautionary tale for all of us who are heading down the AI journey. This is what it looks like to potentially have done it wrong. Let me give you some excerpts here. So after four years of spiraling costs and. That now total, at least. $62 million, a grandiose big data project.

That was a collaboration between MD Anderson cancer center. And IBM's Watson artificial intelligence system is over. The details that emerged in a 48 page audited, our audit report from the university of Texas system. That's surface last week in new stories, MD Anderson, part of the university of Texas system, which undertook the audit over concerns of how the renowned cancer center paid millions. To IBM and other project vendors. DMV Anderson IBM collaboration known as the oncology expert advisor. As a Watson powered clinical guidance program designed to continually ingest patient and research data. Medical literature and treatment options to offer care advice.

According to the report, the ultimate goal of the project is to elevate the standard of cancer care worldwide. According to the audit report. Which paraphrase the project's initial leader and creator. Linda chin MD, former chair of the MD Anderson department of genomics medicine, who is married to the cancer centers, president. Ronald to P Pino, D E P I N H O. MD. The project was proposed to help community oncologists provide MD Anderson quality cancer care to patients who cannot seek treatment directly from MD Anderson physicians. So let's get on to some of the meat here.

MD Anderson's oncology expert advisor is a distinct IBM Watson for oncology product. The oncology expert advisors sought to add more information and wrinkles to the Watson platforms such as proprietary MD Anderson specific patient data. So you have that to consider, but the project has been plagued by huge cost overruns.

The audit report reveals for example, version 1.0, and this is where we'll, we're really center our discussion. Version 1.0, the oncology expert advisor product focused on just one group of cancer patients. That's good. Focus. Those with lower risk MDs leukemia. The original contract term calls for IBM to deliver the MDs leukemia product within six months at a fixed fee of 2.4 million.

Great. Everything seems great. So what could possibly go wrong? However, that contract was extended 12 times. With a total contract fees, a 39.2 million. Other vendors accounted for the remaining money, paid out by MD Anderson on the oncology expert advisor project. The last extension with IBM expired. October 31st, 2016. Over the course of the project, the scope of the oncology expert advisor grew to include five additional types of leukemia, as well as lung cancer.

However, no other forms of cancer, wherever included in the development process. IBM has made it clear that the oncology expert advisor should not be used with patients. And it goes on, this is actually a good article to read. And we're not going to read it in its entirety. As I was looking at this and it came across my feed.

One of the feats I have. And it's interesting because somebody just posted it as a sort of a reminder of what could go wrong. This is what can go wrong in these AI projects. I think we learned this lesson. I think we have learned, first of all, that scope does matter. We have to scope these projects correctly.

We have to organize them around solving problems. And we have to make sure that they solve those problems as quickly as we possibly can. Are they making progress towards solving those problems? The It's interesting. That the project got extended so many times. One of the things I've learned over the years with development projects. Is that you want to see code early and often? And I don't care.

If we're developing something, I want to see their login screen. Like just show me their login screen. Show me they're making progress on stuff. I want to see screens early in often. Are they actually writing code? If they're not writing code. On a development project, then how am I supposed to know that they're making progress? And I've made this mistake several times in my career. And I will not make that mistake again. And I, it, a lot of times when I get with developers who say there's a lot of, there's a lot of code that needs to be written behind the scenes, whatever. And I'm like, No. Now we're D we're doing this in blocks.

We're doing this in bundles. The minimum backend code that you need. And by the way, a lot of that's already written. But right. The middle and back end code you. You can write and then I want to see it working. I want to see, problems being solved along the way. Same thing here. They had it scoped down to a single a single. Cancer strain, if you will. And then it kept growing. It got extended multiple times the project. There should have been clear deliverables after the first. But w whatever the first thing was, and let's see progress.

Let's see what that progress is. Is that progress leading us to solve a problem that we're trying to solve. Sometimes when we scope this too big. And it was a little big up here in the beginning. When they talked about essentially trying to serve. Elevate the standard of care cancer care worldwide. Is that really something. That N D Anderson should be doing. Or should MD Anderson be looking at elevating the standard of cancer care for. MDs leukemia patients. That they serve. And in.

So doing one of the other outcomes is that they elevate the standard of cancer care worldwide. It's really interesting how the scope really does matter on these things. I think the other thing is this whole idea of sunk costs. One of the biggest arguments I had with another leader in at, when I was a CIO. Was over this idea of sunk costs and they had spent. I think half a million dollars.

Now in the scope of things, we were seven and a half billion dollar annual organization. But they have spent half a million dollars. On this marketing. Engine at thing. Whatever it was, and it was a one-off and it did. Some things well and other things not well. And And we were, as it asked to come in and provide them more data and more stuff.

And the problem is. I was the first one who's pushing back and saying, we can't move. We can't move that data into a marketing database. We can't move that data over to that database. And the company that they were using was not, Just did not really grasp that whole concept. You can't have access to the medical data in order to do the things that you're doing. And but we had invested half a million dollars in this product. And I came back and said, I want to kill this project that I want to do.

I want to go in a different direction. That provides for privacy security. And probably an absolutely more robust features than we had. And it was a knock-down drag-out because, how do you, so flippantly look at half a million dollars and. And, just write that off. And it wasn't that I was flippantly writing it off.

I just had identified. It did not serve the needs of the health system. It shouldn't have been selected as a system within our health system. It was not on our strategic roadmap moving forward in terms of reducing the number of applications consolidating into platforms. Creating a platform where the data. We could query the data, but it didn't have to move from system to system.

So there's a whole host of principles and things that we had come up with to say, this is what we want our system to look like. And this system was completely outside of it. It was just other. And there was an act knock-down drag-out. I. Eventually one. That argument, if that's the right word. That we did get rid of that system.

I'm not sure I want it because you ended up with relational debt because that person you have to overcome that. It's from that point on, they're looking at you, like you don't care about the health systems money when in reality. I think when they made the selection process, they didn't care about the health systems money. They just wanted to solve a specific problem that they had in marketing at that time. And those are hard things, but adhering to that, and this is not a conversation about architecture.

It's really a conversation about projects. This project should have been canned well before it reached 62 million. For an organization like MD Anderson. They, they can make bets and lose on bets that are $2 million. $3,000,004 million. But once it gets to. Yeah, I dunno. Once it gets to the 10 million range. Those are bets.

Do you don't want to lose? Those are bets that. Your playing a little fast and loose with things. If you're not sure that you are absolutely heading towards that goal. And you're going to really transform the way cancer care is delivered locally. This should be done outside by the way. This should be done with venture capital.

I was going to say private equity, but no, there should be done with venture capital money. It should have been a third party. And MD Anderson should have from their perspective, probably. Provided them a sandbox to use, but not spending that kind of money. This should have been venture capital money because this is that's exactly what this is.

This is a bet. It's a high risk bet that you're going to be able to transform cancer care. And this is the kind of stuff that gets health systems in trouble. Anyway. So I think my main points here are scope matters. No one to call it quits. Understand the idea of some costs and yes, you've paid this much money, but don't put good money after bad. And sometimes you just have to call, call, project, call the end.

And I think that the big learning here is we are embarking on these AI projects right now. Scope them in a way that they're very focused on a problem set. And do the right. Let's do the rigorous testing around it. To make sure that you're making progress. I just had a great conversation. With Tanya Townson, who's the CIO at Sanford children's and she was talking to me about their. They're testing that they're doing around the generative AI note. And they're looking at it from a lot of different perspectives, obviously, accuracy, quality.

They're looking at it from a time-saving standpoint. They're looking at it from a from a. Physician burnout standpoint, they're looking at all those things. They're not just blindly going, oh my gosh. It works. Look, it does this and this there. There. Right there. And arm-in-arm with the users of a system saying, is this working for you?

Is it not working for you? They're giving them the option to opt out of it. If it's not working. And so there's a testing of the hypothesis all along the way. And the scientific method is your friend, make the hypothesis. Do the experiment, see if it works. And this is and the note is a small scale thing.

They can scale this up to be much larger later if they. So choose to do that. So anyway, I, again, a cautionary tale, the MD Anderson. And IBM Watson project. Just something to keep in the back of your mind. And for some of you're like 62 million, we would never spend that. From a scale standpoint. Your house is something that might be a $2 million project is more than you can really afford to lose. And so you have to scope those things correctly as well. This thing scales up and down.

If you're a large house system. Can you afford to lose $10 million? Probably not in today's day and age. And consider other business models and options for taking those high risk, big bets. All right. That is all for today. Don't forget to share this podcast with a friend or colleague. Keep the conversation going. We want to thank our channel sponsor for today and that's Panda health.

We appreciate them investing in our mission to develop the next generation of 📍 health leaders. You can check them out at this week. health.com/panda. Thanks for listening. That's all for now.