June 24, 2020: Welcome to this episode of the Field Report! Today’s guest is Tahir Ali, from Froedtert Health. Tahir serves as Froedtert Health’s chief technology officer of information technology, providing the leadership and vision to grow the Froedtert & MCW network further through technology acquisitions, leveraging existing technology investments and the use of technological resources. His primary areas of responsibility include the effective management of the daily operations of all technology, including the data center, telecommunications, technical services, desktop engineering, network operations, vendor management and field support. In this episode, we discuss EHR, AI and automation, and disaster recovery, and Tahir explains how Froedtert uses scalability to avoid unreliable networks through unintelligent design. We also talk about on-prem versus cloud, and Tahir gives us his argument for avoiding lock-in, and he also shares with us what he believes will be the most important projects in healthcare coming out of COVID-19. For more on the benefits of AI for both automation and security, make sure not to miss this episode of the Field Report!
Key Points From This Episode:
Field Report: Froedtert Health
Episode 270: Transcript - June 24, 2020
This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.
[0:00:04.5] BR: Welcome to This Week in Health IT where we amplify great thinking to propel healthcare forward. My name is Bill Russell, healthcare CIO coach and creator of This Week in Health IT, a set of podcasts, videos, and collaboration events dedicated to developing the next generation of health leaders.
This episode and every episode since we started the COVID-19 series have been sponsored by Sirius Healthcare.
They reached out to me to see how we might partner during this time and that is how we’ve been able to support producing daily shows. Special thanks to Sirius for supporting the show’s efforts during the crisis
A couple of housekeeping items for you. I’m speaking at the Atmosphere Digital Conference for Aruba, which I just finished recording on the state of healthcare, the impact of COVID on health IT, and the three most important movements in IT infrastructure as a result. You’re going to have to tune in to hear that, go to the Atmosphere Digital, it’s a virtual conference that they’re doing and that will be airing in the next two weeks.
I’m also hosting a panel discussion on The Edge, with Dr. Shadry, the CIO for Seattle Children’s and Rick Allen, the CTO for Naviscent. Healthcare at the edge, the conversions of devices, data, patients, provider and care, which you can hear at the conference and I will also be airing it as a podcast episode. You’ll want to stay tuned to those two things. Now, on to today’s show.
[0:01:28.8] BR: This morning, we’re joined by Tahir Ali, the CTO for Froedtert Health System out of Milwaukee. Good morning, Tahir. How’s it going?
[0:01:36.7] TA: Good morning. Things are going well, busy like everybody else is. But so far so good.
[0:01:42.3] BR: Yeah, thanks for taking the time. You probably have one of the better backgrounds of anybody we’ve had on the show so far. I mean, that’s a pretty cool mural you got back there.
[0:01:50.9] TA: Thank you very much. My wife is into horses so that’s for her.
[0:01:57.0] BR: Absolutely. I want to get right to it with you. I mean, there’s so much we’re going to get the opportunity to talk about. Give us an idea of how your health system experienced the pandemic thus far? Let’s just get there, get an idea of what was going on at Froedtert.
[0:02:13.0] TA: Of course. As with the entire healthcare system in the United States, the heroes were all the clinicians that were you know, out there doing the work that they do best. What we did behind the scene, our technology team, the entire team worked really hard to make sure that we support them. If there was a place where we wanted to get some search beds, we quickly deployed that. If there were some technology needs, we quickly deployed them. When people wanted to quickly deploy work from home, we supported work from home.
It was a really great collaboration between the frontline staff and the back end technology staff to make sure it’s a cohesive effort to deliver the best patient care that we could.
[0:03:00.7] BR: Yeah, we did a little bit of a call prior to this where I wanted to understand some of the things that you guys were able to do. I asked you a lot about scaling up. You talked to me a little bit about a project you had done around your EHR, a significant upgrade prior to COVID. What was the upgrade about and just share a little bit with our audience, what the upgrade was about and what were you trying to accomplish?
[0:03:23.9] TA: Sure, as you know, and I think COVID is going to be a paradigm shift where we do business differently. With the same token, we were a little lucky, we started doing our upgrade on EHR hardware a little earlier to make sure that the scalability is in place. In order to do that scalability and upgrade, we upgrade the back end storage to the fastest storage possible that is currently available NVME RAM. Then we did a very fast interconnect to brand new front end servers. Our capacity, we used to run between 75 to 80%, now we are down to 20 to 25%.
Now, we are at a place where we can scale for the next three to five years without any issues, any kind of different variants that comes our way, we are really ready to take that head on and have that scalability put in. One thing that is extremely important, we got the new servers that have AI built in for specialized EHR workloads. We are ahead of the curve, in order to make sure that if there’s any peaks that come our way, we can smoothly have that scalability built in.
[0:04:43.0] BR: We’re seeing AI come all the way down into the architecture, into the core architecture.?
[0:04:48.4] TA: Absolutely. This AI is especially made for special work loads. It’s customized to a particular workload. If you use that logic of artificial intelligence, you can deliver much better performance with the lower resources. It’s very specialized artificial intelligence building.
[0:05:12.9] BR: What did you end up doing on the storage side and on the compute – and you also change the channels as well, right?
[0:05:18.2] TA: Absolutely. We upgrade the backend too like I said, NVME RAM, the frontend runs our power, power nines that has the AI built in, and of course the interconnect were upgraded to 32 GB times we have eight or 10 different strands. It’s smoking fast, I don’t think you can do any faster than that.
[0:05:39.7] BR: You’re ready to scale? I mean, did – I assume that the pandemic sort of put a little bit of load on to it, to really see how it was going to perform and I assume it did fine?
[0:05:52.2] TA: Absolutely. Of course, as we build more surge beds, the resources were getting consumed, but with this technology in place, we had no issues.
[0:06:04.5] BR: How do you – you worked in New York City during Hurricane Sandy and you shared some of those stories with me, and they’re fascinating. I think that fundamentally changes how you think about architecture and disaster recovery. How – a lot of people are looking at the cloud and looking at different things. How are you thinking, you know, you had this new model in place, how are you doing disaster recovery for this?
[0:06:27.0] TA: Sure, I think with Sandy, it really changed who I am. One thing that is very important is making an extremely cohesive disaster recovery. What does that mean? Making sure that you look at all the peripherals that inject into your EHR, your packs, any business line that you want to do disaster recovery and highly available solution. You want to make sure that you pull all the pieces that build that ecosystem. That’s number one. Number two is make sure that it’s auto failover.
Once disaster happens, there is no time for people to pickup the call, their family, in Sandy people had their homes under water. You can’t ask 16 people to come meet you in some kind of data center, or remotely get to your data center, and do the steps that are required to move from one side to another. It needs to be seamless, it needs to be automated, and it needs to be orchestrated.
Now, fast forward seven to 10 years. Now, the technology’s in place where you can automate and orchestrate a lot of different things through our PA and other automation tools where you can say, if this doesn’t work here, automatically move it here. That is a true disaster recovery for technologists, where it’s a zero touch.
[0:07:56.0] BR: Are we able to do – I know we can do that with some systems but are we able to do that with the EHR yet? Or are we still working through a little bit of the legacy aspect of the EHR at this point?
[0:08:09.0] TA: Sure. EHR by themselves are now built at agent. They do have the integration of saying you can run it here, or you can run it here. They have come a long way. Now, you have to decide all the peripherals around them, even have an active directory fully redundant across two data centers very important. Your active directory doesn’t work, nobody can log in to your EHR. You have to do some due diligence on looking at different things. Some could be more than one and you can have a farm.
The other one could – that does not work in farm, you can have one side, bring everything down and automate everything that I would do in a keyboard through an orchestration tool. They do exist currently.
[0:08:49.5] BR: When you stood up your new environment, did you essentially say hey, we’re just getting two of them, we’re putting one here and one here?
[0:08:56.4] TA: We had two and two. Now we have two in one place, completely highly available within the data center, then we have two more highly available within the data center. At any time, we can move back and forth without any problems and of course, there are some currently intervention required for humans but we are getting to a place where it’s completely automated.
[0:09:17.4] BR: Wow. Has AI built into the chip, I’m not sure why that surprises me. I never thought of that application and it makes so much sense when you think about it. We’re putting AI and networks, we’re putting AI in switches but essentially, a compute platform is a series of networks and switches and moving data around. It just optimizes for the EHR workload.
Are they going to do the same thing for PACS as well, do you think?
[0:09:46.1] TA: I think a PACS has some AI built in currently, but their AI is more on the imaging side and not on a hardware. Now, the newer pack solution and VNA’s are coming out and they can suggest how a tumor might look like or, you know, how a fracture would look like on the screen and provide you some feedback to say, “Hey, I think this is a fracture or this is a tumor, take a closer look,” and then you can – it’s a different AI built in.
It’s more a software AI than a hardware on a chip AI. Yes, it’s going to be everywhere.
[0:10:27.4] BR: Yeah, it is going to be everywhere. You know, I like talking to CTO’s, it’s interesting, the conversation we had before were in a completely different direction, but that’s what happens when I talk to you guys. I love it. Give me an idea of some of the – you talked about some of the design principles butt his whole idea of architecture, you know, we didn’t adhere to it for so many years in healthcare and it got us in trouble. We ended up with thousands of systems, you know, silos of data.
We ended up with essentially unreliable networks because we allowed things to just sort of go without an intelligent design, if you will, of the overall architecture. What are some thing you look at from a design principle standpoint, to make sure that that’s not happening at Froedtert?
[0:11:12.8] TA: Sure, the key now is more scalability across your different application, your different tiers. How do you do that? A normal day, people are sitting there at work and they’re just working and there’s internet traffic. Two months later, COVID happens, nobody is sitting at work and all the traffic is coming from home. The internet traffic, how do you become that agile?
A lot of new technology out there are very software driven. You have to take a look at certain places where you can say, I can define my bandwidth, I can define my resources on the fly. I don’t have to wait for somebody to give me a piece of hardware to add my resources or add scalability to what I already have. If you do a scalable solution and have that scalability through software, you can switch very quickly. If you wanted the SD RAM and you said, “I’m not using this clinic, I’m going to shut this off, and open it up here.” You can migrate that scale, that resources across to all VPNs.
If your VPN is not in use because it’s a Monday morning and everybody’s working, you can migrate that to all traffic that goes to Internet. I think having a great scalable solution, with a layer of software that can migrate your workloads accordingly, will be the solution moving forward.
[0:12:50.9] BR: Yeah, that leads into my next question, which is we were in New York together last year for a meeting, and there was a pretty dynamic conversation about on-prem versus cloud. It was interesting to me because there’s so much hype around the cloud, and there is a lot of practical applications for the cloud but you are sitting there going, “Not for everything,” and you were – and it was interesting to see that conversation. You know what drives a decision for you in one direction or another on-prem versus cloud?
[0:13:24.3] TA: Of course. So again, this is my personal opinion. Every healthcare has their own ways of doing different things. What I believe is cloud is a great place to have a scalable solution. So if you want to migrate something to the cloud and you want that scale and you want that resources that agility on demand, you can definitely go to the cloud. But remember, cloud was really sexy five years ago because their analytics has not matured.
Once data analytics has matured, people started thinking, “Ooh, I put this application on that cloud, that application on that cloud, now I have to pull all of my data back, so I can get rich analytics.” So you have to be extremely careful where you put your data in your application and in your business. Are you going to do analytics? Are you going to mix and match that analytics with something that you have on-prem? If that is going to happen then you have to pull all the data back, and mix it with your on-prem data, and have a richer data analytics.
If you want to move forward with predictive analytics, you’ve got to be very close to the data. The minute data changes, you got to capture that change and see how that analytics change tomorrow, the day after tomorrow, hourly. So predictive analytics can’t – it’s a newspaper versus dot com, so you got to decide what you want.
[0:14:57.8] BR: Yeah, I mean where that conversation ended up going was just you have to understand the benefits of both. One of the things that we ended up talking about there was the need for abstraction layer at the – you know we have abstraction layers between from huge storage and the software now.
We can move those virtual environments all over and that is really powerful but now we need an abstraction layer at the cloud layer, the multi-cloud essentially, and we ended up talking about how important that is because you don’t want vendor lock-in.
[0:15:32.1] TA: Absolutely.
[0:15:32.8] BR: And the people that move this data in – I mean, I read some of these contracts. People move the data in then all of a sudden like, “Hey, we have this really cool analytics things we’re going to do and they go, “Oh well it is not going to work. We just pull the data out," and they realize, “Oh my gosh it is going to cost us almost double the amount to get the data out as it cost us to put it in.”
[0:15:50.1] TA: Absolutely. So now you have data in four or five different places. So you are absolutely right, having a great, great layer where you can containerize your workloads and swing between your cloud, somebody else’s cloud, migrate – Now the cloud providers are also getting really, really clever and they said, “Hey, our cloud has the special one thing than no other cloud.” So you can containerize your workload but still you will be dependent on a particular feature that only one cloud has.
So they are trying to keep you but, as a technologist, you have to be clever enough to make sure that they don’t lock you.
[0:16:33.3] BR: Yeah, absolutely. All right, I want to get down to where the rubber meets the road, you know, so two things I want to talk about: patient experience and caregiver experience. This is the service that comes out of all of these really great architecture that we create. As you considered the patient experience, let’s start there, what projects do you consider the most important right now coming out of COVID?
[0:16:55.5] TA: Sure, so I think not even just COVID itself, but I personally think that we will become more and more mobile. If you look at yourself, your home, your kids, your family members, maybe five to seven years ago, there was at least one PC in the house. Today, there are more iPads and tablets than even laptops. So we used to have a desktop, then it became laptop, now it’s just a handheld. I was looking at a couple of commercial this is, “Oh your next handheld is going to be faster than your computer.”
So as we migrate, we have to realize that our consumers, and our clients, and our patients are also part of that migration. How can you give them everything that they need, that they have at home, same exactly when they come to our brick and mortar? Or if there is a virtual visiting bar. The key is providing them a layer of network or a Wi-Fi that is so robust that they can be anywhere, and it has a better bandwidth a throughput than in their own home.
If we can provision that across our entire network, I think that would be a great, great win. In order to do that, you have to make sure your cabling is up to par. You have to make sure that your access points are top of the line, and you want to make sure your networks behind those IBFs or MBFs are solid and, lastly, your throughput and your bandwidth to the Internet. If you can manage those things and provision great, great SSID through the Wi-Fi, I think that would be a great, great win.
Now let’s say you stay at home and you wanted to do a virtual visit. Nobody likes to do any kind of video that is grainy or that has an issue. That also cleans that up on the physician side or a clinician side. So what we are trying to do is make sure that it is seamless across. It doesn’t matter where you are, it is 100% seamless for your Wi-Fi, anywhere you go. That is where we’re starting.
Of course, moving forward with getting more features inside with 5G, providing handheld, you could bring your own device. A patient could come in and still get all the capability that we could provide them through freight it, that’s going to be next 5G wave that’s coming our way.
[0:19:26.8] BR: Yeah, you think there is going to be more home-based typed solutions, where you’re – I mean obviously, you and I talked about the density. The density in the clinic room is getting pretty significant with Bluetooth, low energy, Wi-Fi, I mean you name it. I mean 5G or 5E. I mean, the density of the spectrum that is being used in those locations, we can also see the same thing happen at home, with the medical center at home really taking off as a result of this.
How are you thinking about extending your – so you thought through architecture for the last decade of within the hospital system. Now we have to think through architecture really in people’s homes. How do we do that?
[0:20:11.3] TA: Sure, so I think you know, we can never and I think it is going to be more and more challenging, how you figure out the last mile. So it’s going to be tough. I can’t dictate what you have in your house. I can’t dictate what type of cellular service you have when you go home. That is going to be tough for us to really dictate. That is also changing. Now when the 5G comes, the cellular service in your home is going to be as fast as some of the Wi-Fi’s that are there.
Wi-Fi themselves are getting better and better, getting a gig in your house is not really unheard of. A lot of people have that. What you have to do is make sure the analytics is built in. You can provide the insight to your clients and your patients, if the experience is not great. How do we do that? Today, we are looking at certain tools where we can say, “Hey Bill, we were having that virtual visit. We saw that there was some graininess. We understand and we can also tell you or show you that how the bandwidth played a part in how your experience was.”
So I think knowing how the experience is going to be delivered is the key to moving forward with better delivery. I think that is where we have to – because you can never the last mile to patients home, and I think we have to figure out how to mitigate that. At least having the sense of where they problem is, is the first step.
[0:21:45.4] BR: Yeah absolutely. Well, it’s interesting I love the conversation. I never – I don’t know why it is just the idea of having systems that have AI built into them, I would assume it is going from one end to the other. We might see that go all the way from the home network into the computer. It’s going to be layered into the entire fabric moving forward. It will be interesting and I guess you are using the IBM POWER9, has that built in.
But we are going to see that probably in the Cisco products, the network, and even other network products all the way out aren’t we?
[0:22:24.0] TA: Absolutely and more and more companies. So they are starting with very customized workloads, very one-off scenarios. They are checking those scenarios, even the facial recognition, the network understanding. Now things exists for security, where this AI built in where they can say hey, this just doesn’t feel right. I won’t do anything but at least I am going to tell you that this particular traffic doesn’t feel right.
So there is a lot of things that are going on that will move the technologist away from touching the keyboard but we will be doing more important things throughout the day than a lot of repetitive things. So I think all around, it would be a win-win.
[0:23:09.9] BR: Well thanks again for sharing your experience and your time with us. You know I think at some point, I wanted to have you and the CIO come on, because she was in New York City with you during Sandy right?
[0:23:24.0] TA: Absolutely, yes.
[0:23:25.4] BR: That story, I mean when you were telling it, it is the kind of thing that nightmares, you know CIO’s have nightmares about for years following. I think it is such a great story in terms of just understanding what can go wrong, and actually just about everything did go wrong in that story, it is really amazing.
[0:23:43.2] TA: Absolutely.
[0:23:44.5] BR: So I’d love to have you back on to talk about that. Thanks again for your time, I really appreciate it.
[0:23:48.3] TA: My pleasure, thank you so much for having me.
[0:23:50.4] BR: That is all for this week. Special thanks to our sponsors, VMware, StarBridge Advisors, Galen Healthcare, Health Lyrics, Sirius Healthcare and Pro-Talent Advisors for choosing to invest in developing the next generation of health leaders. If you want to support the fastest growing podcast in the health IT space, most of you can say this now without me even finishing it, the best way to do that is to share with a peer.
Stop right now, send them an email, tell somebody, “Hey, this is a great show I am getting a lot out of it. You’re going to want to check out these interviews and conversations. The second best way you can do it is you can subscribe to our YouTube channel. We are putting an awful lot of new stuff out there and it is exciting. The live show will only be available on the YouTube channel. It will not be dropped into the podcast channel.
Some of you asked me to drop it in last time but my team here internally wants me to stay focused on the strategy, and that is to have content that is specific to YouTube and to start to diversify our channels, if you will. So some of that, some of our content will only be available on YouTube. So get over there and subscribe so you know when it is available.
Please check back often as we are going to continue to drop shows on a daily basis through the end of June or until we get through this pandemic together. Thanks for listening. That is all for now.