Script from the Webinar. (AI translated - Any mistakes are due to algorithms)

All right, we'll get started. , today's webinar. I'm really excited about this webinar. We have, , two great, a great panel. And I'm really looking forward to the discussion. We have Sean dooby director of services at Sempras and we have Matt Sickles cybersecurity. First responder is what I keep calling you.

What, what is your actual title?

So, , , I'm a strategic architect, so I focus on everything that is security in between. And.

All right. , my nature of doing these webinars is to get to the point very quickly. The title of this is stories from the trenches, how to protect the active directory against ransomware attacks.

, the first thing I want to do is I want to thank our sponsor for today, which is Sampras, , for giving us this platform. To this very relevant topic. And I'm really looking forward to the conversation. We had a, a, , we, we do a pre-call for this, and it's just a lot of fun because you, you guys were incredibly, , forthcoming with, with some of the stories and what you're hearing.

I learned a ton. I was kind of, , terrified as I usually am. When I hang out with, , people who are on the front lines of security. , but as we talk about this, this, , the title of the webinars stories from the trenches, and we're going to focus on active directory attacks. How often has active directory been a target of the attack?

So bill that's the, , that's a great question. , last year, 2021 alone, I was involved in 14 ransomware breach events and, , every single one of them was actually a causative factor on active directory. So those were used as elevated permissions. Those were used as ways to attack the environment. , but a 100% of the time active directory in 2021, we should use on every one of those.

Yeah, my, , my answer is yes. , there is a well-known quote out from Mandiant consulting saying 90% of the organizations that they're involved in, in breach response, active directory was involved. I was on the phone with a very large consulting company, also working on breach response, and they said of the a hundred, , incidents that they had been involved in 99 of them involved, active directory.

I mean, is it because it's vulnerable? Is it because it's, it's, it's the place to go to escalate your credentials is Y

well, , there's, there are a number of reasons. , so th it helps to understand a little bit, , I won't get very deep into this. What, what my executive director unique. So it is, it's been around for coming up on 22 years. So if it were human, it'd be old enough to. And then it terms that's forever and it not only is it still lingering around, it's more important than it has ever been most organizations today for all the talk of the cloud, most organizations are still on premises and if they're on premises, 90% of those organizations rely on active directory and they now count on active directory identities up in the cloud.

So if they're an Azure active directory or they use Okta, or they use AWS or Salesforce, they count on active directory, but it was designed in the late nineties. I don't know about you. I was not the same person in the late nineties that I am today. And there was not nearly as much gray, , So the designers could not predict the, the threats that active directory faces today.

And as Matt is related, as we talked about in our earlier call, it's not designed for security in the way we think of security today. Yeah.

And Matt, you said something in our pre-call that I've been thinking about ever since you said it, which was we haven't, , we haven't rebuilt active directory since the year 2000.

We've upgraded. We've we've done all sorts of things over it, but we installed windows 2000. , and we, we built out that, that forest and that Ady, and essentially we've been working around it for 22 years.

Yeah. It's even worse than that though. Think about it. , you know, we have no other system in any organization that hasn't been, you know, redesigned re.

It has just been, you know, a stalwart, the active directory works. So why do a redesign merger acquisition, divestiture that sometimes will spot it, but think about the chaos that ensues also, you're bringing in other organizations with their bad habits of their active directory and then just trusting them.

So, you know, we show and said, this is an insecure system overall, and we trust it to do all of our authentication and authorization to compute and applications on the.

Yeah, please.

Sean, go ahead. I was just going to say one other factor in why now is that in addition to the natural state of it, and the fact that over time, , active directory has been, , a collection of, , actions taken in haste because they had to get things done.

At, at the same time from the threat actors side, , the tools have gotten exponentially easier to use to attack active directory. So the barriers to entry have gotten very low. It used to be, it was usually sophisticated nation states that were doing attacks like this. , there's a well-known, , security, , person, his name's Kevin Beaumont.

And he said it used to be sufficient. So sophisticated nation states. Now it's teenagers with flame.

That's right. Yeah. You have bloodhound. Do you have other tools that are available freely? You can use those to your advantage and they become script kiddies, right? Those are nothing more than something you download and run against it.

We've all made that terrible mistake of opening up a port on the firewall and allowing eldap connectivity into the active directory for quick authentication from a third. We have all seen that, you know, 2003 to 2010, I can't think of one organization who didn't make that bad choice. Now we are paying for a lot of those bad choices and those band-aid.

You know, I, I think we have to go back to the origin, by the way, I'm going to get to the frontline stories in a minute here, but we have to go back to the origin of this thing. I remember when Novell directory services was out there. Obviously we had other directory services, but there was a big battle between Novell and Microsoft early in the day.

And it was really considered. I remember the conversations back then. It was really considered a place where you're going to put all of your. Right. And all your information about your, it, there's almost going to be a CRM for heaven sake. It was going to you name it. We were going to put it in there and it was going to be a repository for all this information.

Now we've, we've come away from that. But, but that was the design origins of this thing. Is that

accurate? Yes. Yeah. And so I have a perspective on that. I spent a fair amount of time as an analyst and in technology journalist. And so there's an interesting parallel here. So. , as a friend of mine once said, , nobody cares about identity except for identity people for everybody else.

It's just a speed bump. But what Microsoft did to make active directory pervasive through the world is they came up with a killer mail system exchange and exchange required, active directory. People wanted to exchange, they got active directory. That's. Really made the difference. And it's as a side note, that is what's proliferating, Azure active directory right now the two only share the names really they're very different, but if you want office 365 as 90 plus percent of the fortune thousand, do oh, guess what?

You now have an Azure active directory tenant. Even if you didn't know it.

Yeah, and I, I can remember being called, , from HR and asking to extend the scheme and the active directory to keep social security numbers, home addresses. What could possibly go wrong with that?

, all right. Talk to me before we set this up, give me an idea.

If my active directory gets compromised, what does the next 24 to 48 hours look like for me or my.

Most of the organizations that we've gone into post-breach we're post major incident. Then we walk in, we start to help that organization find out where, , you know, everything is Humpty has fallen off the wall.

We've got to try and put it back together. Remember you are going to have trouble logging into almost every device that is integrated with the act of. That can mean networking gear, , tech X radius servers are not going to work. VPN is going to fail. So when active directory is offline, that is one of the core baseline services of the organization.

So the first 24 hours, it's imperative to create a safety net, a shadow of what you have take one of those domain controllers offline, put it in a safe state. So that, that replication won't hit it. Right? Yeah. Sean, I don't know about you, but you know, those are good practices. Anytime there is a major upgrade in incident, , is to put one of those domain controllers in a safe Harbor mode.

Yeah. The idea is certainly is to protect, , protect the replication, , from going around now. There's there's two aspects of that that will get too technical on this, but there's replication. And then there's, what's sitting in the operating system and it does get complicated because active directory is a complicated environment.

One thing that I see when I hold workshops, , on, , deploying our forest recovery. To customers as part of the workshop, as I have them sit down and really think about what their active directory dependencies are. If you're 80, if you're 80 is offline, okay, how are we going to do this? How are we going to do this?

And then they'll, you know, every customer has this aha moment or this oh moment. And they go, oh, well, we'll, we'll store the recovery passwords in our Pam solution. I said, okay, great. How do you log into your so. Oh, right. We'll use active directory for that. , w , what about the, you know, we have to get, we'll go into the machine room and insert a CD and the DVD in the, in the computers, how are you going to get in the machine room?

Your physical badge security system is tied to Azure. Act is tied to active directory. I joke all the time. You need to have, you need to keep a chair outside the machine room by the window that observers the observation window, so that when 80 goes down, you can grab the chair and throw it through the window to get in.

So, let me ask you this, by the way, for people in the audience, , feel free to throw questions into the chat. We got some questions when you signed up ahead of time. So I'm going to throw some, I'm going to pepper. Some of those in here as we move along. , the first question from, from, , one of the signup forms was.

Give us an idea of what some of the different attack scenarios are. So it's not just ransomware, but ransomware is one of them. So what are some of those attacks scenarios? Matt, we'll start on.

And, you know, when we look at those attack scenarios, , they differ, but they are similar. , as we all know, this is fall and file replication system, which is a core and fundamental piece of the active directory since files very quickly across all of the domain controllers.

All of your sites, , is that is a very vulnerable, we're starting to see those common scripted enumerations. We're seeing PowerShell scripts, other things that are, , Starting to be used to attack the environment. , they will go after files, insert the code. , it's not also, , what we saw some of the zero login attacks.

, we're finding that those zero login attacks use a remote desktop protocol RDP to be able to attack the environment as well. So all of these attacks scenarios are based on. Elevated credentials being compromised. And what they're going after is a domain admin, a schema admin, or an enterprise. That has those escalated permissions.

We all must ask for ourselves though, in that environment, should you have that many accounts groups and nested accounts and groups within those permission groups like domain admins, , that should be a least privilege model, but those are the attacks scenarios they're using. , we solved this with, , Conti with RIAA and all of the other methods that are out there specifically focused on.

Yeah, I think that, , you know, there are as to echo what Matt said, there are many different ways of entry fishing, spear fishing, , compromise of RDP protocol. But once they're in the, the, the path tends to follow a pretty, pretty well understood methodology, they do horizontal recognizance, , to gain, , to gain privileged access or what is actually being.

, more common is something called Kerberos sting, which is way beyond the scope of this call. But again, the idea to get domain dominance and once they're there, that's when they make and muck around and look for databases, , exfiltrate data prepare the ransomware, that sort of thing. We worked with an Austin, an Austrian company, a textiles manufacturer that were subject to an attack from zero from zero log-on.

They had not patched their servers adequately. And so. That's an example of a recent security exposure. And if you don't keep up with your patching, then you can become, or you don't fully deploy it everywhere, then you can become vulnerable to it.

Oh, sorry. Go, go ahead. So in what you're talking about is then that weaponization of active directory, there's a lot of information, storage sites and services gives you a map of the network.

You now know the locations of the administrators, you know, where they work, their phone numbers. , there's an organization chart as part of active directory. So not only do they weaponize the active directory, they can use that in. Sophisticated attack.

We would, we would describe that as, not only does it hold the keys to the kingdom, , it also holds the treasure.

And it's the super highway. So, you know, Matt talked about putting stuff on CIS fall than other, , attack vector is good policy. So you can deploy software with group policy and you can deploy it for good or for evil. , and it works. It works both ways. Once someone has perhaps used Kerberos sting to, , gain access to an administrative account that they can do.

, create or modify an existing policy that can spread across the entire forest in a matter of minutes.

So question from the audience, I often wondered this, to be honest with you. What's so difficult about backing up and restoring your after Mo after a ransomware attack, right. Isn't it just a matter of putting it on your TEAC and then bringing it back.

I'm sorry, but putting seriously, just backing it up and restoring it. Ha w w w w why is it so.

Well, there, there are a couple of ass aspects to it. , the, the, the, the most, , the, the first one, let's just call it the first one is that when you're doing conventional backup of an active directory domain controller, what you're essentially doing is you're backing up the whole server, the operating system, right.

And everything on it, the hardware abstraction layer and the active directory stuff is just kind of coming along for the ride. So if you think about the dwell time of malware that sits on the computer before it's triggered, or before you become aware of it, so. In, in an attack sequence, the threat actor goes in and there is under the covers as POS as they possibly can.

And they are in for weeks and sometimes even months in advance, once that means that the malware that's been planted on your domain controllers has been there for weeks or months. And if you restore active directory from conventional backups, then you're restoring the malware. We definitely have people.

Organizations that have been attacked multiple times because they restored conventionally and then the mellow or triggered, or the threat actors triggered it all over

again. Yeah. And simply one of the biggest risks to every organization to answer to the question is when you have single sign on from active directory into your backup system, the attackers are going to go straight into that backup.

They're going to elevate themselves, allow access to modify retention schedules, delete and or modify any of the backups that are there. We commonly see, , the encryption start at the back that upside. So if you have an active directory integrated backup solution, , think about changing that today, making it a separate domain, completely isolated.

If you need a domain or make it local account, because that's one of the attack vectors that are used commonly is to go in and completely eradicate the backup. So then there's nothing even to recover.

Yeah. So this is interesting. It's sort of a regression from what the it pro mind mindset, which is the more active directory integration, the better.

And what Matt is talking about is that maybe not so much, because these are, this is the eggs in baskets

situation. Aren't there set of Microsoft tools that sit on top of this, that protect active directory. And let me know if somebody's doing some things with group policy and other things that they shouldn't be doing.

Well, this is the CIO talking now. It's like don't we have stuff that didn't Microsoft give us some tools around that.

Go ahead, man. I'll let you

start. Yeah, unfortunately, , you know, , some of the power toys and some of the PowerShell scripts that have been provided. Yes, they do give some tools, but it is literally the intractable problem.

Where do you get. I find all of those you'd have to become an expert in writing scripts and writing code. There's not just one download to go put on the active directory from Microsoft to say, let's go protect it. Let's make sure we're using best practice. , you know, the Microsoft security baseline analyzer that was released years ago was a great start at that.

, but you know, we. Other tools that came on the heels of that, like quest and this isn't the early two thousands, mid two thousands, that those were prominent. , but untangling the Gordian knot of what is active directory is a very difficult aspect. , you know, Sean's firm has some great tool sets for that.

, but Microsoft unfortunately does not.

Yeah, I, you know, Microsoft has, I mean, this is not me saying anything and by the way, I'm a 15 year Microsoft MDP. I'm tied to like this with Microsoft micro, but they've moved on. They've they've moved onto the cloud. , they have done basic security enhancements on active directory, but it has been in stable maintenance mode for a long.

Long time now, they, they make, , , they do have a cloud-based product that is involved in active directory security in defender, Microsoft defender for identity. , but the reality is it's, it's been left to, it's been left to, you know, third parties to come up with ways to protect active directory.

Yeah. And one element that is really foundational here. , we were talking about the fact that these have largely been untouched for 20 plus years in some organizations, when you deployed windows server 2000, you upgraded to 2003, 8, 12, 16, 19, et cetera. All of those versions require an update to the.

Either a native mode or a mixed mode, you are the most secure. If you have the most modern Microsoft version of your schema and your databases, but very few organizations have that they run in hybrid and legacy mode for compatibility purpose.

Interesting. So w we have questions that keep coming in, keep sending your questions in.

I didn't introduce myself. My name is bill Russell. I'm a former CIO for 16 hospitals system out of Southern California. So. The we're going to break this conversation down into before the attack, during the, the attack and then after. Okay. Cause I want to get a, a picture for, , for what people can do. And one of the questions, , we very small on my computer here.

We we've seen some tools that monitor a D however, you only find out about stuff after it. It. Isn't there anything that acts more proactively to actually stop or slow down and attack. And that gets into my first question, which is what can we do? This is before an attack. What can we do today? What's the pragmatic thing we can do today, , before we could compromise.

Well, it's critical opinion. No, Matt, I actually I'd like you to go first before I.

Where we are falling down is while we log all of these events that occur with an active directory, , our logging management and our security operation centers are the protective bakes. If they don't have the use cases, the definition of risk within their alerting, their monitoring, and then their response that's going to cause a problem.

, the event quantity that goes into an active directory is. , being able to weed through that. , so we have to fine tune those. You have to build your use cases to find out. Is there someone trying to log in unsuccessfully to a domain controller? Do those accounts that are in privileged groups? Do they get their password changed at a higher frequency?

Is multi-factor authentication used. , I could argue that multi-factor authentication may be one of the greatest prevention tools, , as a whole that could be used against the active directory. , and then also managing and controlling service accounts. If you take those elevated permission account.

Put them behind lock and key. You have a break glass account for extreme situations, and then you make sure that you are managing and monitoring through global managed service accounts, through Microsoft functions that are modern. That is a great preventative thing, but those are just best practices. The majority of organizations are not following.

I would. Do both say the, both the macro and the micro and the macro to what Matt had talked about. Matt mentioned surface counts and wow service accounts are really a thing. So, you know, if you're not necessarily an ADA expert for most everything that you have a service account is just a regular user account that is you market to the password to not expire usually.

And then you use it to power, a service like SQL server or. Other things that are available in active directory that through Kerberos sting is one of the most popular methods for compromising active directory. Right now are people that have service accounts that are, have weak passwords because a threat actor can easily crack, , , a weak password offline.

And I won't get into the Kerberos syncing. , They make these service accounts, they give them privileges. So you find that the threat actor finds a service account. , they it's very easy because as Matt said, active directory is open to enumerate all of the user accounts that have service principal names on them.

They pick some, they crack the passwords of the easiest ones. And if. Elevated users they're done. They've already within minutes. W our lead, , incident response person said his record for cracking an active directory. Once he got inside was 10 minutes. So that's the micro for a service account. So go forth and find your service accounts and give them long hearing gnarly passwords.

Number one, , and the broader, the broader. Is evaluating your ADA attack surface, as we've been talking about it's accumulated the cruft of 20 plus years, and this is where we provide Sempra is provides a free active directory security analysis tool called purple night. And purple night, we'll evaluate your active directory for 80 different indicators of exposure or potentially indicators of compromise in minutes.

, and folks like Matt, , with Sirius and other folks. And it's, it's a brilliant tool and we encouraged no matter we encourage you to run this tool and, and, and find out, find out what your exposures are and learn how to mitigate.

All right. So you're going to give us no longer than a three minute demo of what you would see from this tool.

Matt, I'm going to ask you questions and there's going to be rapid fire. They're coming from the, from the audience. So

when you want me to start and you can set the style, you

can put it up on the screen. And Matt, I'm going to ask you to answer these questions. , what are the best practices for backing up active directory?

So make sure that you don't forget about the system state restore that is built in to every domain controller. , put a manual process in place, , grab an encrypted USB drive, rotate that out, put it in a vault to save on top of your normal backup procedures. This is going to be your safety net in case something goes horribly.

All right. Number two quick answer. , why do traditional monitoring and recovery tools within Ady? Why are they not sufficient?

So one of the biggest things that Sean was just hitting on there was that when a password is gained for like a service account, with an elevated permission, it looks like normal activity.

If you're sharing the service account across the domain, reusing it for elevated permissions running system services that doesn't look like malicious activity. Once you have the password, it's very difficult to find. Malicious activity until something is planted some type of payload, a dropper or some type of level removement begins.

, last one, before we get back to Sean, I've seen some papers on restoring active directory, DCS and forest. However, I do not understand. Why it takes so long to restore why?

Well, , in a perfect state, , you would be able to go into the room. You would be able to put the USB drive, recover it, , depending on the size of your NTDs, , your, your directory service data, , It's large.

, you're, systole all of your file. Replication has to be put back together. , if you have a, you know, a solid backup method, , you can recover, but once you get the primary domain controllers back online, holding the main roles, it then has to propagate and it has to replicate out everywhere. So if you have 50 domain controllers, you may get a complete wipe out of those 50, you have to restart.

Primary locations and then bring all of the others back online as well. It's not just a single server. Right?

All right. So, , Hey, I linked to this, , purple Knight tool is in the chat. , I'm going to ask Holly to go ahead and post that link. Once again, it's a little up in the chat. She can post it again and then, , Sean, take it away.

Show us what this free tool is going to show us about our activity. Okay.

So, , this is, , purple Knight is a free download. Like you see there in the chat. It, when it unpacks it all unpacks into a folder called purple Knight doesn't require any SAS aspect. It doesn't phone home. To Sempra is it doesn't do anything.

It stays entirely on the PC that you've run it from. , it does not require any rights in the domain to do this investigation. It doesn't even require any rights on the PC to do it. I wouldn't, , if you run it, I would recommend that you notify your SOC, that you're going to run it because it's running, , a series of tests that may look unusual.

It's just purple Knight, that exi, and every time you run it, you get a license agreement. And I say next. And then it finds the forest that the D that the client is joined to. In my case, the forest is called a four dot lab I selected. And then I say next, and then it shows that it's going to run 80 10. , 79 are selected.

So it has the zero log-on tests like that incident that I talked about, but it takes a little bit longer. So to run it quickly, you would run it first without the zero log on. And then you could go back again and run it with the zero log on. So I say, run, you see it going through the tests it's already gone through 20 of the tests takes about 30 seconds.

And what the end of this, it will give you a score card of. The, in order of priority of the things that you should look at and the things that you should do things with. So you were a 75 and, , we, we also have, you know, it'll also check for updates and of course, good things like that. And it's finished.

And I can't an F in my environment, , a lousy score and it tells me right off the top. And the different areas, account security. I get an F group policy, Kerberos security. I get an F I can zoom in on a full report that shows you in great detail for anyone to understand why the score is and what it, what happens.

It puts the most important ones right at the various. This is evidence of a bad guy, inserting a backdoor into active directory that the event logs won't show here's problems with constrained delegation. Here's all sorts of here's principle. Or this is print nightmare. Someone's set up for print nightmare.

You've got privileged users with weak password policies, and then. A further list of what is wrong. What's going on here. And you can see, for example, you can check out any of these, let's say right here, this well-known privilege said, this is the footprint, all the DC shadow attack. And showing that there's a user called bad actor in the environment where they are and what remediation steps should be done.

These are all mapped to the ANSI or the MITRE attack. There you go under three minutes

under three minutes. All right. So how do I stop your screen share? There we go. We stopped your screen share. That was before. How quickly does it take to stand that up? I mean, that ran, we did all that in three minutes.

I mean, if I have a large active directory, is it still three minutes,

Matt? You've got some, some, , boots on the ground experience with that.

Yeah. And we've been using this tool, , to our advantage when we walk into a client that is under duress. , we have seen, you know, the three to five minute runs. , that's a good sign that, yes, there's some misconfigurations, there's some improvements.

, but unfortunately in a poorly designed, poorly configured, active directory, , we've seen it take a couple of hours, , that is a leading trailing indicator that says, Hey, you know what? We've got some bigger problems here. , I do want to highlight one. Sean did not have any elevated permissions to get all of that intelligence on the domain.

While this tool set is a gooey. That gives a great report. There's a lot of downloadable tools that give you even more intelligence about an active directory domain with no elevated permissions. This leads right back to that concept. And that compliment we made at the beginning active directory was never designed to be secure.

And it is designed for the E information sharing between computers and people.

All right. We've been compromised. So we're moving to the next phase. This is we, we, we had our chance, we missed our chance. We've been compromised. How many of these attacks lead to the organization having to rebuild some aspect of their active.

, almost every, , we're, we're seeing a lot of that, especially if the dwell time, , is protracted. So if we find that the attackers were in for several weeks or months, , that is going to require a redesign, it's going to be an. To the current environment. , we also see just complete damage and destruction to the active directory that requires manual rebuild, especially if the backups, , are lost, , in a mutable backup of your active directory has to be there.

, or you will have to redesign and rebuild a large portion. Remember one of the brands and playbooks from most of the incident response of cyber security insurance firms is to cut. Objection to the public internet and therefore you don't get any remote help. So your skills on the site of active directory, those are probably not going to be prevalent.

You might have one or two people who have that skillset. That is a real problem. Yeah. You,

you were saying that there, there are cases where you're on the phone, telling people the PowerShell command and they're typing the PowerShell command in, and then reading back to you. What's going on. I mean, that's the level of.

Of of disconnect that has to happen in the, now that we're in the event. It's.

I can, I can remember it event last June. , we were on the phone literally for eight to 12 hours on a FaceTime call and there was a screen that was up in front of us. And we were watching all of the commands come back locally on a domain controller, and then we were on speaker.

Giving what to type so that we could get some intelligence. We could get some of the help. They had lost all of their backup. They had nothing, but the domain controllers that were online, we manually export it objects and then had to redesign, rebuild the active directory, which was two weeks plus.

Yeah, I, I really, , a common, , defense or incident response techniques.

Is when, so companies like, like Matt and Sirius will bring us in as active directory experts. Cause we've got, we like to, we like to joke that we have the biggest collection of active directory experts outside of Microsoft. And I think probably inside of Microsoft after all this time, when we come in, one of the first things that we try to do is take a backup of active directory with our active directory forest recovery product.

Pull Aidy out before the threat actor, either on a normal course of operations or because they sense there's an incident response going on crypto locks, the environment. So get a copy of active directory out where it's safe and restore it in an isolated network. Like Matt's talking about when, once it's in an isolated network, Number one, you know, that you've restored that then that the active directory service itself has no malware on it.

Then you do an analysis of what's inside active directory to see with a tool like purple Knight. Or we also, we have a variation that is a post-breach tool that shows you, oh, look, someone's been added to domain administrators, someone. I injected, , Sid history to get them so they know what steps to take.

So you can formulate an action plan without the threat actor, knowing what's going on. And then you design the threat, the action plan and execute it all at once and hope that you know, but if worst comes to worst, you still have a valid copy of active directory in areas, slated investment.

Yeah. And even after, after eight to 12 hours, people get tired, people get exhausted.

, inadvertently, , we were doing your recovery with another client, , and instead of moving 30,000 records, they got deleted. And you know, that that happened.

, let me ask you this. So we've, we've shown some of the Semper stools. I didn't have Sempras beforehand. So I didn't have those advanced monitoring tools.

We get compromised. Do people still go back and can you use the separate tools post a post-breach pro compromise

mat.

Yeah. And as Sean was mentioning, there are different flavors of their tool sets. , they have a great incident response team that we've taken advantage of and a lot of our clients face.

, and as they've been brought in, , not only do you get that, , you know, that cursory view. Of your report card. , but as Sean mentioned, you get to go deeper, you get to get some of those, , exploit details of what happened when they happen. And you can start to see some of that stair-step , and Sean, I'll let you talk about some of your teams, , IRR recovery, but I loved your comment at this point.

You probably do have more experts in your team than Microsoft.

Well, , and, and we have often found, you know, we've been talking about this, Matt is that in over 20 years, active directory is not the sexy technology it used to be. So the, the skillset, , involved in of the 80 practitioners, there are fewer 300, 400 level 80 practitioners, which is what it takes to dive into the.

To repel the invaders or to do something like a forest recovery, for sure. But we, , we've had situations where we've come in and we see sometimes, and, and this is the thing, and maybe this is something to take to heart. Is that it, the, the threat actors are not necessarily, they're not doing these solar winds types attacks.

If you think about it, I mean, that gets the press, but the, but the reality is. If you put yourself in the mindset of the attacker, they want to get to the database. They want to exfiltrate the data they want to get there. The straightest way the they can to maximize the profit. They don't want to say, oh, I'm going to try this novel technique.

Or I understand the north Koreans did that. No, they're going to go in and, and look for the lowest hanging fruit insecure, , service account passwords, you know, all the things. PK the purple night shows shows you to remediate. So a lot of times just this basic, you know, seeing what's wrong, fixing what's wrong and in an incident response, you'll see, you can see the, the, the, they're not necessarily that sophisticated what's been done.

And, and so it is possible to climb out of something like this. But as Matt was saying, before you started changing the live environment, you have to have. Your backups are 80 and your environment, not your backups, your restored environment. Ready before you start jacking with what you have.

All right, I'm going to do, let's do three more questions here and, , I'm just going to read them as they are.

Are there any recommendations for protecting against the malicious use of group policy to spread ransomware besides limiting permissions?

Yeah, limiting permissions is a great aspect. , making sure that you have a catalog of your group policy, making sure that you're going through the health of them, , monitoring for any newly created group policies, those are all best practice.

, Sean, your thoughts.

Yeah. So our director of product is basically the foremost group policy expert in the world has been known as a group policy guy forever. , and so our directory services protector. More broadly, it takes the, the basics that purple Knight has purple Knight is a one-shot deal. Just, just does that.

DSP runs the, the security evaluation continuously, but it also tracks changes real time and active directory. So for example, a group policy. So any changes that are made to group policy, , you can be notified about, and you can do a comparison between one group policy setting and the previous version of the group policy.

And it will show you. Very clearly what's been changed. And so, for example, if a malicious modification has been made and all you have to do is click a button to roll it back to the previous version of group policy DSP does that for active directory in general, it monitors through a patented process.

Any change that is made to active directory in an untamperable way. It doesn't matter if the threat actor tries to obfuscate tries to erase logs, whatever directory services protector sees, all of the. And you can roll things back manually or even automatically before the operator even sees it.

There's a, there's some RPO RTO questions.

I think people are scratching. I sort of scratched my head from time to time when I hear, you know, health system down for 35 days or 40 days. And I, and I think that's the nature of where these questions are coming from. So let me just give them to you. Is it necessary to have an on-prem restore solution?

Is there any Azure cloud-based solution where we can back it up and if necessary can be used to raise to have it as a restored.

Matt, do you want to start it?

Yeah, last week I had a client, , , they ran into that same scenario. , that was actually one of their saving graces. , they actually had in Azure, a protected domain controller.

, they were also using Ady connect to sync. , so there are some best practice that are available for Microsoft 365 instance. , but. To the multi-master replication. Once it changes on prim, it's actually going to roll out to the Azure instance as well, because it is all the same database. Once files are manipulated and they are encrypted in the Cisco and the, , you know, in the file, replication service, that's where the problems do.

But we're going back to that original. You must have a backup to be able to recover. And that is what is being tampered with. If you have a valid backup and you have that ability, you're going to be able to roll back and get that things back up.

That's what I would say is it's the backup. I mean, Azure is, Azure is great.

, I'm using it. I use it all the time, but it's still what's on the backup. It's great that it's in immutable storage. If you put, if you put it in Azure blob storage with a notable policies, so it can't be CryptoLocker. Great. But when you restore it, you're still restoring the operating system and the malware that's on the operating system.

And this is, , yeah, it's it's it's interesting. And the next question, I keep hearing two weeks to rebuild a D if there, if there's a, sorry, just move to, if there's a complete unrecoverable disaster that can't be acceptable. Isn't there a tool or set of processes to bring the RTO and RPO down.

, Sean, go ahead. Go ahead, man. That's that's what active directory forest recovery. That's what the tool is all about. Active directory forest recovery brings the RPO and RTO down. We're seeing empirically an average of 90%. Of what it would be. Otherwise, if you even have a disaster recovery plan, a cyber focused disaster recovery plan, I would ask you as in the audience to go to your people and ask if they have an active directory disaster recovery plan that is really focused on a cyber disaster.

I have talked to a lot of companies and a lot of very, very major companies. I've been caught with their pants down on this. , so do not assume that you have a cyber protected, , active directory disaster recovery plan

and run the tabletop. You have to go through a tabletop. You have to make sure that this is actually something that you practice.

If you're not including that in your tabletop, you're not going to decrease the return to operation period.

All

right, last one, before I go onto the next thing, how old is too old for an 80 backup?

, I'll take this one for starters, 80 is different than other, other, other utilities. , and well, we typically recommend when, so when we, most of our customers have a recovery point of 24 hours active directory, , can be restored, , within, , No with the recovery objective, sorry, I'm mixing up my terminology, but Ady beginning to can be restored so that it's no less than 24 hours old.

Some companies do 12 hours, which is really pretty remarkable. , what we find is that. , I just completely forgot your question. I ran off the rail. How old is too old, right? Right. So we find most companies take one backup per day for the first week, and then they will keep one back. Keep one of those backups.

They're all full backups. There's no incremental as a differential. Then they'll keep one backup once a week for a month. That's things start to get really ugly after a week and active directory for reasons that we don't need to go into, but you keep these older copies for forensic reasons. If you wanted to find out, if you wanted to recover a month back to find out, if the threat actor had already made changes into your environment.

Yeah. And I'll, I'll just say that no backup is too old. If you lose everything, , you know, if you have to go back six months and that is the only one you have, , you have nothing else. That is the only copy that is actually going to get to something. What we've talked about those two weeks, you know, that 35 days to get back up and running.

, yeah. Even if it's six months to a year old, you can still use that. , you're going to have machines that can authenticate, they're going to drop off the network, but, , that is, , you know, that's, Love's labor loss. You have to move

forward. Well, let me ask you this. So we're now after, after an event has happened and you guys have been associated with, , organizations that have gone through these events, , I'm going to assume that after an event, people have a different perspective on things.

What kind of things do you find that people see differently after they've gone through one of these events? , Matt, we'll start.

Yeah, they have no idea. , what all of the connected systems are to active directory that rely on it. Phone systems, badge, access, readers, you name it, , parking lot gates. , everything is connected.

, if you don't have a data workflow of what is contingent and reliance upon the active directory. , make that part of your incident response and your disaster recovery plan, because that's the biggest lesson learned is they didn't know the importance of active directory. They had taken it for granted for so long.

It has just been in place for years upon years, potentially decades. And that is what they don't have visibility in.

I would say that once you've been to the crucible of incident response, , it, of course it depends on where you starting from. , one of the things that or people learn is it's technical, it's technological.

So from a technology point of view, people have become very religious and using active directory forest. , to, to run disaster recovery drills on a quarterly basis, because it's very easy to do with the product to set up an isolated network, move the backups in there and do it. Then it becomes more and more of a, just a process of regularly doing business.

The other aspect of it is that people neglect and I actually I'm Schilling for my podcast here. I just did a podcast recording, talking about the organizational aspects of. , incident response, the things that people don't think about, what my guests would talk about, tearing open the envelope and saying, okay, for the duration of this incident response, this is the reporting structure that was agreed.

That was agreed upon beforehand. And you will follow it because there's no time for infighting and putting off decisions and all that when you're in the middle of an incident response is I'm sure. Knows exactly what I'm talking about, the difference between a quick incident response and a slow one, the organization has a lot to do with it.

Yeah. It's , I mean, we hear these incidents stories over the years, and I'm always reminded of the two lane when a Katrina hit and two lane in this story was they didn't plan on the fact that they would not be able to get back into their building and everything was in the. And today we don't think about that.

Cause everything's in the cloud. We can get to it remotely, no big deal, but all the backups were in the building or at least they thought they were in the building. So they couldn't actually physically get to the backups to do the things they needed to do. And you know, and those stories, we've all heard those stories over the years of just, oh, we didn't think this active directory.

Connected to so many things. Cause it's so convenient for us. We think, oh, badge access and it's all connected in this great, isn't this phenomenal and the things that make it easy to use also make it very, , critical when it, when it's, when it's lost. Let's get pragmatic here in the last couple of weeks.

One, I want to look at different timeframes and you just tell me off the top of my head, what we can do. , what can someone do today to protect their active directory from an attack or from ransomware specifically? , Sean, we'll start with you

run purple night and do what it says. If you need help, if you need help consult a partner, if you don't have the expertise to get it done.

Yeah. So, , if you're a leader in the organization, if you are a member of an organization who's responsible for technology, identify who the primary and secondary active directory resources are. And in the next 24 hours, your call to action is to go find them and ask them the question. How do we get our active directory recovered?

If we lose our backup system? That answer to that question will tell you if you have a confident level of recovery or a non-confident level of

all right, I'm going to give you a little bit more time to play with that was in the next 24 hours. That's the action item. , the next quarter, the next three months, I got three months to do something.

What should I be doing in the next three months? Matt will start.

, highest recommendation, make sure that your backups are truly immutable. Make sure that you have air gap copies of critical systems, not just the active directory. , validate that to ensure that you're not just thinking that you have a good copy or a good backup, run a tabletop and make that tabletop reoccur with curiosity, do it on a quarterly semi-annual in an annual basis to make sure that you know exactly how to recover that.

And when something occurs, it's not the first time you're doing it. , this is the evolution and the maturization of your incident response and BCDR plan.

, I would say evaluate active directory forest recovery, look at what it has to offer. Look at how it can protect your active directory against ransomware attacks and give you the ability to restore without having any malware on it.

Being able to restore in all kinds of situations from on-premise you can restore to the cloud risk score from VMware to hyper V many other many other capabilities and do it very, very fast. To immutable storage, as Matt says. And also what Matt said, it makes it much easier to do quarterly tabletop exercises because as anyone that has worked in it and made and worked in it, operations knows that when the time comes for you to make changes, you have it scripted out ahead of time.

You don't do it from the seat of your pants and it's even more so in the case of incident response, you, that's not the time to be thinking dreaming up. How do I do. Yeah.

, I'm going to give you a little bit more time. This is the last one I'm going to give you more time. What, what should I be doing this year?

I've got a full year. I get, I get some budget. I get some money. What, what are what's something I could do this year that, , that is going to protect my active directory from, from an attack or ransomware? , Sean, we'll start with.

Okay. So we've, we've talked about the reactive, the recovery aspect of it.

So you have a little more luxury of time. Let's talk about the proactivity part of it. This is, you know, if you're looking at the NIST cybersecurity framework, you know, protect, detect, respond, and recover. So let's work earlier in the, in the NIST framework and let's look about the protection and respect.

From our viewpoint that's directory services protector. So you then continually monitor, monitor all of your active directory vulnerabilities. So PK will show you if you've got something there, if an administrator has been added, whatever, but if you want to know if something's been has happened in the, between one and the other, and you want to be alerted.

That'll that'll tell you. It'll also allow you to see all administrative changes, all user changes and all potential threat actions that you can then literally roll back with one click.

Oh fantastic. Matt.

Yeah. And coupled with Sean's recommendations, , focus on protecting those privileged accounts, , privileged service accounts as well.

Multi-factor authentication is not a guarantee, but it is one of our best protections we have against ransomware events. , making sure that the, , you know, least privileged model is kept at hand and you were doing the bare basics in your domain, admin, your enterprise admin group. There shouldn't be someone in the enterprise admin groups on a routine basis, admit them as an only necessary purpose.

Make sure your permissions in your active directory are modern. , and most importantly, again, look at multi-factor authentication as one of the best protective methods we have in the industry right now against attacks for elevated and reuse of credence.

Gentlemen. I want to thank you for, I was going to say sharing your time, but mostly sharing your expertise with us.

, during this, during this, , webinar has been fantastic, , in the four and a half hour in the three, some odd hours I've spent with you guys. I I've learned a ton. If I were a CIO today, I'd have a whole set of action items just from this conversation. And, and I really appreciate it. Thanks for your, for your time.

And I want to thank Sempra. For making this webinar possible, and this will be available via a playback. We're going to take this recording, put it out there and you can continue to tell people, Hey, I listened to a great recording and they can hit the, , the website register and watch the entire recording.

, gentlemen, again. Thank you very much.