This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

UnHack (the Podcast): Security Consolidation and the Viral Google Bug with Chris Plummer

Introduction

Hi, I'm Drex DeFord, a recovering CIO from several large health systems and a longtime cyber advisor and strategist for some of the world's most innovative security companies. And now I'm president of This Week Health's 229 Cyber and Risk Community. And this is Unhack the Podcast, a mostly plain English, mostly non technical show about cybersecurity, and RISC, and the people in process and technology making healthcare more secure.

And now this episode of Unhack the Podcast.

Drex DeFord: Hi everyone, I'm Drex and this is UnHack the podcast, and with me in the virtual studio today is Chris Plummer from Dartmouth Health. Hey buddy. How's it going?

And Oh, same here.

Drex DeFord: I think in a lot of ways the Mutual fame club here you're super famous. I feel like I know you because we have messaged back and forth several times. But I listened to you on Ed Gaudet 's show risk Never Sleeps and you guys had a great show. And so for folks who are listening to this, if you haven't listened to Ed's podcast, you should, especially the one with Chris.

because it really is like you are in a pub and you're just kind of sitting at the table next to a couple of people that are just kind of goofing on each other and talking about a bunch of, I mean, really obscure stuff you guys go down the road on like music and a bunch of other stuff, so

Chris Plummer: that's what we planned to do.

I think after the first time Ed and I talked, which we were mostly talking shop and then we vowed we would talk again. And then there might have been 10 minutes of actual healthcare cyber talk in our second episode, but it's worth it if you can just hang in there.

It's stitched in.

But let's start, I wanna see what you bring up about your background, and maybe we will riff on that a little bit. You're not the kind of guy who woke up at 11 years old and said, oh, I wanna be a chief information security officer, or I wanna be involved in cybersecurity. Like that wasn't a thing, right?

Chris Plummer: Yeah. So cyber was not invented when I was 11, of course. I've always had an interest in computers which goes way, way back to like elementary school when I learned how to sort of game the education system in my, my both elementary and junior high. By plowing through my work as fast as I can.

And so, that sort of just love of technology has always been with me. Which maybe that's this need to want to protect technology is sort of how I've come to wake up every day and do what I do now for hospitals. Yeah, no security was even my first sort of serious infrastructure role, which was right out of college.

Sort of being the technologist for a .com. That was a fun time in the nineties .You were standing up all this infrastructure to support this frenetic pace of business and so we kind of secured it. I think we did. The thing that was apparent to me is that I didn't know anything about security drex.

Like I knew that I was creating future problems. And so, yeah. We were really lucky in that organization. We had really deep connections to GTE at the time. That, that's not an entity that's around anymore, but we had some managed security services through GTE, which I found to be very necessary.

Trouble still finds me just like it finds anybody else, right? There's nothing you could do about it, but, yeah, really wasn't until I went to work for in support of US Navy. For a little over 10 years I was information systems and security manager. Yeah, so

Drex DeFord: We've got some overlap there.

I'm a retired Air Force officer, we did DOD stuff for a while. That's a good time. Right? You enjoyed that?

Chris Plummer: Yeah. I, if anything, you know exactly what you're supposed to do, Drex. I think that's the moral of working in service of the government. It's not ambiguous, you don't love it.

I truly burned outta that job. mentioned this too before other conversations, but probably like three times where like serious burnout where I have a hard time getting outta bed. Yeah. I just, I can't will myself to go work on certification and accreditation activities, which

i'm still very involved with that.

Chris Plummer: being like the only, being the only one Right. Which just seems to be like a role that I continued to fall into for a long time, being the sole chief technologist.

It probably for like three or four jumps, after that role. It's just exhausting, bearing, all that operational responsibility and then the responsibility of securing something and then bailing the organization out when something goes wrong. And then fast forwarding to my first work in a health system, I was the only cyber FTE for a number of years.

And then I came here to Dartmouth Health. Where, I'm fortunate to no longer be the only cyber FTE, It's an amazing change of pace. In this country's 6,000 plus hospitals, that's very much the state of affairs, right? Even if they have a cyber FTE at all even a part-timer.

it's a huge spectrum I think, as well, yeah so

Like literally those kinds of conversations that just show the broad range of resources and lack of resources that are some of the challenges that I think we're all facing right now as a country.

Chris Plummer: Yeah, and that's, so as a country is an important phrase. improving that status quo is why I became a member of the health Sector coordinating council now, like five plus years ago.

Knowing how hard it is to fight in the trench for an HDO. Yeah. I had relationships. In federal government and federal agencies where I knew if I could be inserted to help inform federal healthcare cyber policy, I could make life easier for myself and many others. Which is something, we in the council have been tirelessly working to do.

And so, this was a consolidation effort that Navy Marine Corps underwent. Where at the time, and this is like 15 years ago or so. Every naval installation was sort of defending their own perimeter. They had their own independent network stack. The stuff that was running here locally I worked here at sub me activity, which is submarine maintenance activity here in Portsmouth.

They're running network equipment that was sourced like locally from here, a company here in New Hampshire like that stuff's not in San Diego. It's not in Pearl Harbor. And so, among, a lot of other inefficiencies Navy Marine Corps decided, hey, it, if we corral everyone through a common stack and it's not gonna be one hop away, right?

And so I feel like DOD can't speak for other aspects of DOD but that corner figured things out really fast. And I've always wondered why is that model not exist for national healthcare? Why are 6,000 plus hospitals still like waging independence, fighting for

Drex DeFord: the independent, fighting against nation states, right?

Chris Plummer: Yeah. And it's not even pride, it's not like, well, I want to do that. I want to have my full control. It's like, you know, I'm, I'm up against powers I can't fathom at a scale that I can't fathom. So that's maybe an aspiration that through the work we do

in HSCC and hopefully other, groups, like-minded groups that we're driving toward an objective like that where we. I liken it to this marvel, you're gonna get tons of Marvel references for me. because that's just what I do I see on the wall behind you. But this concept of Ultron, which was this like suit of armor that Tony Stark wanted to wrap around the world.

A number of clinics that are the size of small hospitals. So, as we encounter these new relationships with other organizations, I'm helping to put this suit of armor around those organizations and those of us on the security team here at DH work incredibly hard to make this work.

Because when we combine forces with other organizations, the security piece can happen fairly quickly and efficiently. And I also feel like it's just such an incredible bang for buck. Of course the business operations, clinical side of hospital, like that's a years long integration effort.

But the security side of things. Lots of hard work, but can happen super quickly and it just puts that suit of armor around them that you wanna put around everybody. It really

Really quickly. That is a good feeling too. It's great for patients and families. It's good for the people that work there. It's good for, they're referring patients to you and you're sending them back home.

Chris Plummer: I feel that lift off of their conscience too. I really do. When we were able to activate certain.

Aspects of our defense and I feel really good about that. It's exhausting for the business to continue to affiliate with new organizations. That whole, model of healthcare that we're in right now, it's a whole other conversation.

You know, whether it's, is that how healthcare should work in America? Should it just be this progressive growth of health systems? I don't know how else it works, at least from a cyber perspective, Drex especially the advent, we're gonna, use the tired phrase, AI over and over here about, yeah, let's talk about

Drex DeFord: AI.

It's not what's going on with AI.

personally, I think the state of affairs for AI and cyber right now is it could make an awfully good. The word copilot is interesting becuase Yeah, I'm with you. That's been branded, but

Drex DeFord: yeah.

Speaker: an AI copilot for a cybersecurity analyst, let's say, could be incredibly powerful. Something that guides a decision making process, but is not entrusted to making decisions.

So I think that's where AI is at right now. Where we could reliably at like, of course we all have that, that dream that it's just on full blown autopilot and turn it over the

Drex DeFord: AI and let it go.

Such a deep understanding of the user, like understanding identity. Who is this person? What is their role? What are they really practically doing right now? Right. To inform a choice about taking an action on the system they're using or the account they're using. I just think that level of enrichment is so complex.

Drex DeFord: we start where we can start. Part of this too is I feel like. Let's see if you agree with this. There's a little bit of an unrealistic expectation for AI in that. There are things that we do today that we suck at, and then we add AI and we expect AI to suddenly be perfect.

But it, I don't feel like it has to be perfect. I think that's all part of the learning training process too.

Chris Plummer: It's hard. 'cause on one hand the institutional knowledge required is so deep, but the rate of innovation is so fast. I can't pretend that in a year from now, like some of these problems won't be solved.

Some of this stuff happens and it feels like it's in a blink of an eye, so, we started using

Drex DeFord: GPT. When did we start using Chad GPT? Two years ago? Yeah. I've been paying

Speaker: for it for two years.

who they are at all. And they're running through a laptop farm that's run by a US citizen, laptop farmer, somewhere in North Carolina. So they've got a local IP address. How far it's come so fast is amazing. And then just in the last couple of weeks we've had the Google VO three stuff come out and, you can make motion picture quality

movies just what prompts It's crazy.

Chris Plummer: The arts will be a hard sell, at least for me, like I love Broadway, for example, like that is a live show that's happening right in front of you that you will not replace that, you'll not synthesize that with ai. So maybe that's why I like that so much.

I've been to many shows in New York and it's like, I can't get enough of it. Maybe it's the reality portion of that experience in the face of all of this automation and AI that we're sort of fighting off in other parts of our life. I do agree with that sentiment.

So it made all these assertions, but it's so confident. Drex, it's so confident. it had this license plate it was, okay, so the car was an Audi Fox, it's a wagon. It's a station wagon from the seventies. It's super cool looking. And and the license plate said Foxy, which is like, okay, that's clever.

And so. Chat GPT was convinced it did not say that at all. It like doubled, tripled, quadrupled, quintupled down on what it thought it was, and it was flat out wrong. And then I said like, I'm gonna bet you a million dollars, your answer and my answer. And then so it, like, for the sixth time, it, like, it went in again, it was absolutely wrong.

Drex DeFord: I I feel like, there's a dial, I think there's a dial in there that we have to be able to tune and like, you need to be a little more humble.

It's possible that you could be wrong.

Chris Plummer: Humility. Yeah. I'm still happy to use it for troubleshooting though. Like, as I was saying before, like providing a series of, let's say like five steps, five things to check if you're trying to solve any given problem, because I do appreciate that about it. If for nothing else, I like throwing random problems at this thing, that's not gonna judge me for asking you questions. No matter what it's about. And it's gonna be overwhelmingly pretty positive. It's attitude is never really gonna change. I do want that for us, but I do, I want that multiplication of force for our analysts though, of course, because signal volume is it's so large and there's no practical way,

Speaker: you know,

Chris Plummer: we'll never staff our way out of it.

They bought their own stuff, they sourced it locally. It's amazing. They bought 12 PCs at a time and they paid a premium because they never knew when they were gonna get those PCs replaced. And I pulled all that money back to the major command that I actually started buying things centrally and we were able to get

so much more stuff just by doing that and standardizing and securing and doing those kinds of things. But we do it in our health systems today there's gotta be a way to scale that to the, you know, this is, we're like thinkers in that we've had this experience in our past, but we just can't get to kind of the, I don't know, a med net or something like, Sure.

Where all hospitals connect to this thing and, but there's some, when you're connected to Med Net, you're at least this tall to ride the ride. Yeah. And it provides some level of reasonable security and connectivity. I don't

Chris Plummer: know why we wouldn't want that. Drex. And this is alluded multiple administrations, right?

Yeah. That's why I'm still in the seat, hoping that day comes and then we all, I dunno, that's the end of the movie for us. I don't know.

Speaker: Um,

Chris Plummer: Congratulations, you got it done. we have so many other just structure.

Speaker: I mean,

Chris Plummer: Even like breach notification for example. Let's say something has gone wrong.

How do we all know something has gone wrong? Every time we hear, this pervades the news. This pervades the information products we consume another hospital suffering a security incident or perhaps we know it's ransomware, perhaps we don't know that. The first thing we all do as HDO is scramble to understand what's our risk profile look like Now that we know that this potentially.

Drex DeFord: Maybe we know, maybe we don't know everybody. Lawyers up too are kind of like a lot of, and I understand that. Pulling their horns. Yeah,

I get, it's frustrating, but I cannot help but feel like. The H ISAC, for example, which I think is a great representation of where, all H dos are at, from a intercommunication perspective. We all ask the same exact questions in H ISAC every single time. Some hospital is suffering, like, yeah, what's going on?

Am I concerned? Do I need to do one of the rinse repeat five things that I do every time someone's breached? Do I turn off my VPNs with them? Do IS remote access? Do I cut off their email? If they have local accounts, do I shut those off? Whether sort of, interfaces Do I have within the sending medical telemetry? Do I have to cut those off? Is it a third party?

I think the legal piece of this is always going to exist, but there's very much a patient safety aspect to this that I think needs to take some precedence over the legal characteristics of the situation such that hospitals can know.

I can continue to keep my conduit between my EMR and their EMR going. Everyone desperately needs to understand that state of affair every single time. Like we are all searching for those same answers. It consume so much oxygen in the room. Just trying to understand.

So we're all doing the same thing we patrolled news sources, we're looking at social media. We're trolling Reddit. Just it's wild. That this is

Drex DeFord: what we're doing to try to get the intel that we need to make decisions like

Chris Plummer: that. Yeah. It's another illustration of hdo is all still on an island and fighting their own battles.

And again you're going to get ahead in the information war without people. So you're just, you're losing on all fronts if you don't have the personnel right? You're not securing your house like the windows are still open you are not getting the information you need

about your third parties. A big part of

That's gotta be a big part of it too, instead of doing it all serially.

Chris Plummer: Yeah. I mean, So regional information shares can help in that. We have a pretty good one here in New England. We might be 50 members strong, I think at this point, I'm not sure. 30, 40, 50. Of course the hi sac.

Yeah. Which only Earl Weiss can tell you exactly how many HDOs he accounts for, but an awful lot of them. Yeah. We lobbied for that too in sector coordinating council, just getting some kind of universal access to an ISAC for HDO is because that is such a powerful information share and,, it's not that. H ISAC, for example, is expensive. It's an entity that has operating costs. It needs to be subsidized, and federal government doesn't subsidize the ISAC. But still it's a budget line item for a critical access hospital that's barely making anything work.

I don't think there's any other practical way to do it,

Drex DeFord: And there's a lot of tough decisions at those level of organizations too, right? Do I buy an EDR? Do I repair the leak over the emergency room? Do I replace the broken MRI? A lot of these are like, there's a no win situation going on it.

A lot of these facilities.

Chris Plummer: Yeah. Do I run the 20-year-old MRI still producing images that are pretty good? Yeah. It serves our purpose. Yeah. But carries a lot of baggage that I can't see and I don't understand. Boy, I wish we had some more optimism about the situation,

Chris Plummer: I was on my way to karate class, which I've been a martial arts practitioner for not quite 10 years now. Yeah. But I'm a second degree black belt now. At the time I was training for my first black belt and I was on my way out the door to class, and I got this really peculiar email at the time, Google had just rolled out this feature, this sort of blue check mark of authenticity. For the

Drex DeFord: senders

Chris Plummer: yeah, can imagine there were like 2 billion Gmail users. The spectrum of ability is incredibly wide, and so they had this really noble idea that they would use this standard called BIMI

Which is underpinned by a couple of other email standards to basically assure you, the user this email really came from the place you think it is. You're not about to get scammed by clicking this link or, calling this number, so on and so forth. So anyways I have this.

Drex DeFord: Uhhuh.

Chris Plummer: And it was from UPS. And the message had this UPS logo in it, and a place that the sender can't put it, it actually appears in the Gmail UI and I was like, this can't be right, but I've been wrong before, so I'm going to class, I'll deal with this one. I come home and I just like, I can't stop thinking about this thing all night. And so by the next morning I'm sure this is a bug.

Drex DeFord: Yeah.

Chris Plummer: Which is a crazy thing to even come to terms with because you're like, did I just find a bug in a Google product?

Drex DeFord: They're also smart and there's millions of them. How can?

Chris Plummer: I'd never fought that fight before. And so, but by that morning, like I woke right up the next morning and I like just threw it all together and submitted it to their bug tracking system. And probably by the afternoon they were like, yeah, this is working is designed, this is not a bug.

Drex DeFord: fight this battle for a while, right?

Chris Plummer: To get there? Yeah. But I had friends though because that night I went to Twitter back when Twitter was really sort of at, probably at the apex of Twitter or getting there.

Yeah. And I was venting on Twitter about it. I get tons and tons of support. And then it was this amazing, cascade of attention to this bug. And it got Google's attention google changed its mind like the tweet was at like nine o'clock at night, and it must have been sometime the next day it reversed course.

And you kind of almost broke Twitter because of

Drex DeFord: that, right? Wasn't it?

Chris Plummer: It went wild. It was like every. Media outlet on Earth was covering this story, and it was in like Apple News, so it landed on every iOS device in the world and it landed in some Microsoft News applet that appears like in every Microsoft operating system.

Speaker: And,

Chris Plummer: I did interviews and such, and I was in, but I was in the local paper, which is the proudest thing and my family could see that. Did you tell your

Chris Plummer: Yeah. I was on our local news and my daughter was down there at the local news station with me that Saturday morning.

But yeah, that was amazing. Not just 'cause of the media focus, but there was a substantial infrastructure change made by Google because of this very quickly. In fact, they turned that feature off like for the rest of that week. It was off while they tried to figure out what the heck was going on.

there were also a number of other cascading failures. Like UPS for example, had a serious flaw in their infrastructure. They changed that very quickly. Microsoft did Yahoo, did apple mail, did like, these are things that came outta the woodwork in the months after. It was an incredible series of things that were, this sort of shook outta the tree from this.

went to a party once locally and they're like, Hey, are you the Google guy? And like,

Drex DeFord: we're almost out time. I'm gonna ask you about one other thing 'cause it just happened recently. So the switch from Twitter to Blue Sky and then you just posted something on Blue Sky that kind of went viral too, right?

Yeah, I saw it. The big surprise like, Hey, I didn't realize that was happening when I,

Yeah. But I was lucky enough to catch early showing of Mission Impossible. And so, as I was just sort of scrolling through news and there was a teaser for a news story about whether or not mission Impossible had a post credit scene. And so, like any good nerd, I first analyzed the page in URL scan to understand all the HTTP transactions I would be signing up for if I clicked this link to learn something I already knew.

because I watched the movie and you watched the

Drex DeFord: trailer.

Speaker: Which didn't exist, which it wasn't a thing. There was no trailer. But it was like an overwhelming response to how dirty the web has become. And and then there are not a lot of great analytics in blue skies to understand like, why did this explode?

Why did this go crazy? Why has this been seen? It's been reposted, I think almost 5,000 times and liked 15,000 times.

But you have no way of really knowing why it's amplified like that. It just, it caught wind. But I think it really just told me like, no one's happy with the state of the internet these days. Like I grew up with the internet of the early nineties as many did. I've seen a huge swing of evolution. I was really lucky to help author the very first textbook on the internet at the University of New Hampshire in the nineties while I was still a student, which was crazy to walk across campus and kids are like carrying a book that you helped.

Right. My wife who I met at college actually took that very class with that textbook and I was very helpful in her homework, but she was like, I know that

Drex DeFord: guy.

Chris Plummer: Like I wrote that homework, but I wrote a visceral reaction to the way the web works and the way the web is commoditized.

Drex DeFord: are so. I feel like we've maybe crested a hill here a little bit where people gave up a lot of privacy using the internet, using social media, and maybe now we're starting to, as a public, understand how much privacy we've given up and how much advertisers and the government and everybody else knows about it.

Sounds super paranoid. I am a little paranoid too, but that's the nature of that. Maybe that's what, maybe that's why it's gone so viral, is that people are starting to look for that kind of information, like it's actually triggering.

Speaker: And I think they're exhausted too. I think they're exhausted at the kind of countermeasures that you need to employ to maintain some kind of privacy and so good.

And I think it's clearly generational. We did not see this until, I don't know the z is it the Z who are most vocal about this? The Gen Zs? I don't know.

Drex DeFord: Unintended consequences, I think of some of our, early internet actions and implementations.

Chris Plummer: but

Speaker: that,

Chris Plummer: that world. Drex is very crafty. Yeah. And they will find a way. Web advertising is such an incredibly oh, I don't know how to frame it in a positive way, but

it is fascinatingly complex. They will find a way. They're incredibly brilliant people who work on these things that we very much don't like about our Internet experience. They will find a way to overcome whatever we think we're gonna throw at this problem. So, that's just another one of these eternal struggles I think we're in for.

But that was a really fun, that was like my first big. The blow up on big blue sky. Big blue sky. So yeah, we'll see where it goes. Hey, listen,

Drex DeFord: um, I'm out of time. I'm disappointed in this, but I am come back, do this again at some point down the road. I've

Chris Plummer: pages and pages of things to talk with you about that.

It would be so fun to do it again.

Drex DeFord: I would love that. Chris Plummer from Dartmouth really appreciate you being here on UnHack the podcast.

Chris Plummer: It was my pleasure, Drex. Thank you so much.