It’s never good to get a call at 3 a.m. – especially one informing that systems are down. And yet, when it happened to Zach Lewis, CIO and CISO at the University of Health Sciences and Pharmacy in St. Louis, his first thought was that the aging infrastructure was the culprit.

As it turned out, it was much worse.

What Lewis had actually woken up to was a LockBit ransomware attack; one that would consume days of all-hands troubleshooting, trigger calls to the FBI and the organization’s cyber insurer, and force a high-stakes negotiation over a $1.25 million ransom demand.

Zach Lewis

During a recent UnHack the Podcast episode, Lewis – who authored a book based on the ordeal – offered a glimpse into what a ransomware experience looks like from the inside, and how leaders and staff can prepare for the unexpected. 

Not “an IT Problem”

One of the most impactful (and terrifying) parts of the ordeal is that, for the first few days, nothing pointed to a security incident.

“We didn’t have a ransom note,” Lewis said. “We hadn’t heard any alerts or messages,” including from EDR and SIEM. “All indicators were that it was an IT problem.”

That, he pointed out, isn’t a failure of process, but rather, a reflection of how modern ransomware attacks unfold. The environment came back up after the initial troubleshooting push, then went down again.

In fact, red flags didn’t become apparent until Lewis and his network director got into the root of the ESXi hypervisor that managed the organization’s virtual machines. At that point, they noticed strange file extensions and a README file that’s “not typically there.” Upon opening it, they found the ransom note. 

Weak Link

Another compelling aspect of UHSP’s experience is that the organization wasn’t unprepared – quite the contrary. In fact, the cybersecurity program had scored at an A-minus level by external attestation. His team held briefings with the board, established connections with CISA and the FBI, secured cyberinsurance, and conducted tabletop exercises.

And yet, LockBit got in through a gap created during a firewall migration.

According to Lewis, the organization had moved configurations across three successive firewalls. Somewhere between the second and third transfer, a VPN access control was lost. It looked correct on the surface and was labeled the same way it had always been, but the limitations on who could connect to VPN was blank, meaning anyone in the environment could connect.

“It’s funny how just a couple things have to line up at just the right time,” he noted. “If we had checked a config a little bit closer, if we had done this one thing with VPN, we would’ve broken the kill chain, and the whole thing would fall apart.”

It's the kind of gap that doesn’t surface on a scorecard or a point-in-time assessment. The lesson? Finish projects completely before moving to the next one. "Just take a minute and review your configs,” he said. “Did things get set up right? Are they properly installed? That would’ve saved us a lot of headaches.”

The 1.25M Bluff

When the ransom note appeared, LockBit claimed to have 75 gigabytes of UHSP data and was demanding $1.25 million for its return and deletion. What followed was a negotiation that Lewis described as one of the most important variables in the organization’s outcome.

A key strategy? Buying time. Negotiators were able to do that by asking for file listings, requesting samples of the exfiltrated data, and keeping the conversation flowing for as long as possible, which in turn provided more time to assess the actual risks.

The downside is that as the negotiation stretched on, LockBit’s claimed data volume kept growing, from 75 gigs up to approximately 380 gigs, which raised immediate questions. “We have no idea where that number came from,” Lewis said. “We don’t just have that sitting anywhere.”

When the organization ultimately declined to pay and LockBit dropped the data at the deadline, the actual volume was two and a half gigabytes. The file listings that LockBit shared during negotiations turned out to be the full extent of what they held.

“They were bluffing the whole time,” he remarked. Although the files contained some possible FERPA data, there was nothing of significant sensitivity, particularly from a HIPAA standpoint. 

This outcome, however, isn’t necessarily a typical one. “Another company could have done that and it could be two gigs of sensitive research information or government information,” which would likely prompt a payment. “It really depends on your situation and your leadership team and what you think is in that data.”

Ransomware as Big Business

Throughout the conversation, Lewis returned to a theme that still catches many executives off guard: ransomware groups like LockBit don’t operate out of basements. They operate out of office buildings.

“People literally come in and they clock in, they clock out,” he said. “They have benefits. They have quotas of how many companies they need to attack in a month.”

LockBit, for instance, develops and licenses its malware to customers who execute attacks and share portions of ransomware earnings. It functions, Lewis observed, like a franchise model in which leadership, HR functions, technical development, and quota structures are all present.

It’s a model that isn’t easily broken, he said, noting that LockBit didn’t dissolve after its public-facing sites were taken down by the FBI. Instead, it dispersed, regrouped, and began reconstituting itself by pulling together affiliates from other disbanded ransomware operations. “There are no extradition rules,” he said. “They stay in the wind.” And the underlying infrastructure – which includes server farms housed in buildings overseas, outside the reach of U.S. law enforcement – moves with them.

Leaders, therefore, must be laser-focused on trying to prevent attacks, while also being equipped to act decisively if they do happen. 

“You have to stay focused on the task at hand,” and be willing to forego sleep while guiding teams through the darkest hours. Doing so isn’t easy – for Lewis, it meant spending time away from his wife and two young children. But by putting in the extra time, he was able to help his team weather the storm, and position themselves better for the future.

“A deeper level”

For UHSP, the incident did that by uncovering gaps in data governance. For instance, during the negotiation, leaders couldn’t answer questions about what exactly LockBit had, where it came from, and how sensitive it was.

That ambiguity drove a multi-year data governance initiative that is still underway. Files are now classified, documented, and tracked, according to Lewis, and data security posture management tools flag files containing HIPAA data, credit card information, and other sensitive categories. As a result, the response team can answer questions definitively and quickly if exfiltration happens again.

The university also hardened its identity and access posture, moving toward passwordless authentication and biometrics where possible, and tightened controls around browser-based access to SaaS applications, where most of its data now lives.

“We found out where our weak points were during the incident,” he noted, and were able to “get down and understand our data at a deeper level.”

And hopefully, be able to respond even more effectively going forward.

Zach Lewis is CIO and CISO at the University of Health Sciences and Pharmacy in St. Louis, and author of Locked Up: Cybersecurity Threat Mitigation Lessons from A Real-World LockBit Ransomware Response.