How are our problem solving skills? Today we explore this as a deficiency of Health IT.
Today in health it Friday ramblings, first principle thinking and our deficiencies in health. It, my name is bill Russell. I'm a former CIO for a 16 hospital system and creator of this week health, a set of channels that events dedicated to transform healthcare. One connection at a time. We want to thank our show sponsors who are investing in developing the next generation of health leaders. Short test artist, site parlance, certified health. Notable and service. Now, check them out at this week. Health. Dot com slash today. Hey, we've put a new service out there on our website. We aren't launching it yet, but would love for you to take a look at it. And provide any feedback or input that you can this week health.com/news is your one place for health. It news let's face it. You really have to Wade through a morass of content in order to find the best content. That is relevant to health. It, this site is curated by health it professionals. So the articles are only those that are relevant to you and your job. In fact, if we talk about a new story on the show, You're going to find it on this site. Check it out today and send me feedback DME on LinkedIn or Twitter. Alright, one last thing, share this podcast with a friend or colleague use it as foundation for daily or weekly discussions on the topics that are relevant to you and the industry. They can subscribe wherever you listen to podcasts. All right. X, formerly known as Twitter is actually a pretty volatile topic right now, especially for those of a liberal bent. I've had conversations over the last couple of weeks with people who are very disenfranchised with what Elan is doing. Has X changed since Elon took over from my vantage point? It's an accurate depiction of the dialogue that is going on in our nation. It's crass. It's raw and in many cases it's ill-informed and I don't think that's really changed. It's a. It's a town square. This is what happens. If you post something about mother Teresa, you will invariably have somebody who says really nasty comments about mother Teresa. It's just that kind of place. And it really always has been. So has Xchange. I don't think it has, but that's not the basis for our discussion, the basis for our discussion. As a post on X from the X engineering team.
Love Ilan or hate Elan. You can't knock his engineering chops. He's running circles around the big three in the automotive space. In fact, in a rare, honest moment, I heard the Ford CEO talking about. The difference between Tesla and Ford and he was talking about their control systems and in the average Ford card, there's 130 different control systems. The software's written by, I don't know, let's just say 70 different companies. So when he wants to update something, he has to go to those companies, get it written. And, , they release something that goes out to their cars. Whereas in Tesla's case, it's all written by one company.
It's one control system, one set of software, and they can literally do updates. On the fly, could do updates daily to your car and I've already gone out. I have a 10-year old Tesla. You can go out there. AN update can come down to your car and I can end up with a very, a new instrument panel. They could do that level of of adjustment to the car, which is pretty amazing. And you could see why they're running circles around the big three. Another instance he's made Boeing and NASA and everyone else who's trying to do space exploration, look like they're standing still. In terms of rocket development and those kinds of things. The boring company, which if you're not familiar is those company that makes the drills, that would make things like the Lincoln tunnel that go under rivers and that kind of stuff. And gadget reported that the Prufrock too, which was unveiled in August of 2021, could dig up to a mile per week. Think about that a mile per week underneath the river. It's pretty amazing. However their success or the Prufrock three is planted. Dig. Up to seven miles per day. And it's changing the way people get around Las Vegas. I commented on this, that we went on the loop while we're in Vegas and there. Building tunnels around Las Vegas and using Teslas to get around. It's just really interesting. You can check that out. The only place I'm waiting for Elan to make some progress is in his words, turning the largest fusion reactor, known to man the sun into renewable energy that is in enough quantity. To change how we think about power. And I'm going to come back to acts at a minute. But that whole. Thing was just to say it's really hard to knock as engineering chops. And what he's been able to do with the companies that he's formed. But before we start talking about ex. Let's take a look at his approach, which of course is not original with him. He just happens to be the best modern example of utilizing this method. And that is first principles thinking. First principles thinking is a problem solving and reasoning approach that involves breaking down complex ideas and problems into their most fundamental parts. And then rebuilding a solution from the ground up. The concept is rooted in physics and has been used by various thinkers, innovators, and entrepreneurs throughout history. So again not original with him.
The main idea behind first principles thinking is to avoid relying solely on analogies or past experiences, which can be thought of as reasoning by analogy instead. You should identify and analyze the most basic foundational principles or truths of a problem, and then reconstruct a solution from those base truths. Steps for first principle thinking are identifying, define the problem, clearly state what you're trying to solve and understand. And then break it down to the fundamentals, dissect the problem or idea into the most basic components, ask questions. What am I assuming? Or how does this fundamentally work? Then you reconstruct the problem using the fundamental truth. You've identified. I think of a new, innovative way to address the problem or create solutions. And then finally iterate. You don't start with perfection in mind, but you iterate to perfection, test your new solutions or ideas, refine them based on feedback and continually improve them. And the example of course, is electric cars and batteries. One of the, the most. Famous examples. Again comes back to Elon Musk and their approach to creating cheaper batteries for the Tesla electric cars. Instead of accepting that batteries were just inherently expensive, because they've always been that way. Musk and his team broke down the battery problem into its fundamental components like nickel, carbon, et cetera. Then they look for ways to source or manufacturer each component more. Cost-effectively. Leading to a significant reduction in the cost of producing. Batteries. And you know that the benefits of this are innovation. A clear understanding of the problems and being able to build solutions around them and then avoiding the status quo bias, which I think is very prevalent in healthcare. And that's how I'm going to tie all this back together. While first principles thinking can be a powerful tool. It is also more time consuming than reasoning by analogy, and it may not always be necessary for every problem. However, for complex challenges, especially those where traditional solutions aren't working. It can be an invaluable approach. So let's get back to X and tying this back to healthcare. There's a post on X today. October 27th. And I'm going to read some of it for you. And it really detailed some of the things that they've been able to do in the engineering side, on X over the past year,
so here's what this post has to say. This has been a full year of engineering excellence that sometimes can go and notice besides all the visible changes on the app. Here are some of the most important improvements we've made under the hood. And I love this because we get to see. Behind the scenes.
First consolidate the tech sacks for you. Following search profiles, lists communities and explore around a single framework. Simplify. We may, we heard BJ more just drive this home in an interview recently. Simplify. Reduce code simplify. Anyway, second thing completely rebuilt the for you serving and ranking system from the ground up resulting in a decrease of 90%. In lines of code from 700,000 to 70,000, a 50% decrease in our compute footprint and an 80% increase in the throughput of post. Scored per request. Think about that 700,000 lines of code to 70,000 lines of code. Anyway, I'm going to go on unified the four U and video personalization and ranking models, which significantly improved video recommendation, quality refactor, the API middleware layer. Of our tech stack and in, so doing simplified the architecture by removing more than a hundred thousand lines of code and thousands of unused internal end points and eliminating on adopted client services. You remember when he was doing this, when he was like, reshuffling the deck over there, everyone's oh, it's going to crash. They're firing too many people. They're getting rid of people, getting rid of people that aren't good at their job. Isn't going to impact you all that much. And I've plenty of stories. To that effect, I would have spare you on those stories. But at the end of the day, a team of five great developers can replace 500. And I think you're seeing that be the case here. Anyway, it goes on reduce post Metta sourcing latency by 50%. And global API timeout errors by 90% blocked bots and content scrapers at a rate of 37% greater than in 2022. On average, we prevent more than a million bots signup attacks each day. And we've reduced DM spam by 95%. Again, these are perfect numbers. There's still going to be bots out there. There's still going to be DM spam and that kind of stuff. But they're making progress in that space. Shut down the Sacramento data center and re provisioned the 5,200 racks and 148,000 servers, which generated more than a hundred million in annual savings. And total refried up 48 megawatts of capacity and tore down 60,000 pounds of network ladder rack before re provisioning it to other data centers. And again, you cannot argue with Elon Musk's. Commitment to sustainability in computing and in all other areas. Optimize our usage of cloud service providers and began doing much more on prem. This shift has reduced our monthly cloud cost by 60% among the changes we made was a shift of all media blob artifacts out of the cloud, which reduced our overall cloud data storage size by 60% and separately, we succeeded. In reducing cloud data processing costs by 75%. Think about that reduced cloud data processing costs by 75% significant savings.
It goes on built on-prem GPU, super compute clusters and designed, developed and delivered 43.2 terabits. Of new network fabric architecture to support the clusters scaled network backbone, capacity, and redundancy, which resulted in a $13.9 million per year. Savings started automated peak. Traffic fail over test to validate the scalability and availability of our entire platform continuously. So as you would imagine, X is a target. And you could very easily. See why? Because they're a target. They have to constantly be worried about redundancy, resiliency, security, and those kinds of things. All right.
So what does this have to do with healthcare? I'm afraid we have more buyers in health. It, these days than innovators. Buyers biotechnology. And in many cases take credit for the great thinking of the developers as proof that they are innovative. We see this all the time. We're innovative. We implemented nuance, Dax, copilot. Innovative is actually a stretch. You're a buyer was some organizational change management skills. That has implemented someone else's innovative work. I've said in multiple settings now, Judy Faulkner has made more bad. CEO's looked good in healthcare. Then really deserve to look good. Epic is innovative and Judy is brilliant and epic. First strategy is an efficiency play, not an innovation strategy. Don't get me wrong. You can't out innovate the market. You have to rely on partners. No one should go it alone. If I were an epic shop today, I might implement a mostly epic first strategy, but that's the point we have to dissect and understand our problems better in order to lead better. Stay with me. Cause I know I'm saying some hard things here. Let's take a look at the ex post again, but this time, think about your health system. Or your clients, if you're with a partner and consider, if we have done this type of work, Here, the work here, the thinking here, the expertise here, the numbers, it's really amazing consolidated the tech stacks. Can we do this? Have we done this completely rebuilt a whole series of the software to decrease 90% of the lines of code? Could we even do that? Do we have those development skills? Can we measure it in that way? I love the measurements in here. They're very definitive. You can see the progress. Unified, different services refactor, the API middleware reduced a hundred thousand lines of code reduced the post Metta sourcing latency by 50%. Blocked content scrapers and those kinds of things. It's shut down the data center. Reprovision 5,200 racks, 5,200 racks. And , this is a year essentially. Since I think it's about a year. Maybe a little more than that since Ilan took over and complete a pit people happened, but they've been able to in that chaos, Reprovision 5,200 racks hundred 48,000 servers. A hundred million in annual savings in total freed up 48 megawatts of capacity. Do you hear that kind of things? Again, I come back to, here, the work here, the thinking here, the expertise here, the numbers. We have a significant amount of waste and inefficiency in health. It. The CIO job in healthcare was one of the hardest jobs I've ever had. Actually it is the hardest job I've ever had. However, the part that was the easiest for me was finding hundreds of thousands. If not millions of savings. Every year based on poor architecture. Bad thinking. Poor processes, which. Quite frankly, they just seep in to health it over decades of time. There were so much of it. It was heartbreaking that we really couldn't move faster as a team to pick up more of the money every year.
I will say this. This kind of thinking requires engineers, technicians, developers, and true it professionals. I love the fact that physicians are stepping into the CIO role. But I caution that you will need true technology. Partners. You're going to need architects. You're going to need leaders. Especially if this is not your background, someone that can consolidate tech stacks create software when required, see design patterns, refactor, middleware, design data systems, implement sustainable models for massive computing. And storage as well. Optimize the cloud build on-prem systems when it's appropriate and scale the network for redundancy and resiliency. And oh, by the way, do all of this with an eye on security. Are these types of skills present in your current it team? If you run a billion dollar plus health system. They really should be there. If not all on staff, then your partners, your consultants, you need to have access to this kind of thinking these kinds of skills. These are the skills that are required to handle the pace of change in healthcare. And to deliver the kinds of results that we are going to need.
Some of you are hearing this and you're thinking, oh my gosh, we have none of that. Where do we start? And the answer is first principles thinking. Identify the problem. Break it down. You get the picture, start with that. As always reach out to me. If you have questions that's all for today, don't forget to share this podcast with a friend or colleague. And we want to thank our channel sponsors who are investing in our mission to develop the next generation of health leaders, short tests, artists parlance, certified health, notable and 📍 service. Now check them out at this week. health.com/today. Thanks for listening. That's all for now.