But the Red Team!

It's happened to me a few times now, and if you operate or have had an audit performed by a third party Red Team, I'm sure it's happened to you. Every time I hear the phrase, "but the Red Team..." I want to...well...respond like this:

It goes something like this: during the normal course of your work in information security, you discover countless programatic and technical deficiencies. You find gaps in patch management, failures to follow defined processes, gaps in access controls and visibility, legacy systems creating risk, assets missing security tools or configured out of compliance, poorly deployed and maintained tools, the list goes on and on. As a practitioner charged with defending this environment and closing the gaps, you have to convey to your leadership a clear definition of the problem, what it will take to solve it, and your progress in doing so. At the same time, you have to be very careful about your message, lest you be viewed as that "sky is falling" security guy who just wants to unplug the Internet. You do your best. Priorities will be selected amongst the sea of problems, some you agree with, some you don't, but you get to work and time passes. Then, incidents happen, priorities shift, staff changes, opinions change, new requests are added etc., further complicating your ability to effectively close the discovered gaps in a measurable and programatic way. Thus progress is slowed, although you are doing a ton of valuable work - the work that is being asked of you and described as your priority. You are making progress on closing the gaps, but all the complicating factors make progress really slow.

Then it happens. In secret, your leadership team launches a Red Team exercise as part of their own due diligence to obtain a 3rd party or somewhat external perspective on things. You may or may not be aware until the final reports are in, and those results will invariably be that the Red Team was able to quickly and easily find a vulnerability, exploit it, gain a foothold, move around the environment, and capture some bright and shiny trophy all without triggering a single alert. They will present their findings to the security organization, pointing out the ease of effort and short time it took for them to capture their flags. They will map their actions to the kill chain to make it look very adversary-like in approach. Big words will be dropped to mask the simplicity of what was actually done. You'll sit there the entire presentation nodding your head in agreement as the results confirm your own observations. "Yep, knew that was an issue." Leadership however, will be shocked, furious, disappointed. They'll ask questions like "why was this so easy?" "why didn't you detect it?" "why didn't you tell me it was this bad?" "didn't you say you already fixed that?" You'll be more shocked by your leader's response than the report itself...stunned into silence, not knowing what to say.

Then, the Red Team leaves (as far as you know), or begins scheduling the next offensive.

Next comes the new set of priorities, and the new mantra on everyone's tongue and measuring stick for every new initiative will be "well the Red Team found..." or "that's not what the Red Team said," or "but that's what the Red Team did." All of the sudden, your expertise doesn't matter. You prior observations, recommendations, and progress are tossed out the window because, "the Red Team..." The junior and inexperienced staff in the organization will look upon the Red Team with awe. They will be heralded as some uber l33t hack team that saved the company from being the victim of not knowing just how bad things were, and celebrated as the savior who finally brought clarity to what really should be done. Everyone will chase the Red Team report, scouring over it to find what must they do to fix the issues found, or to seek blame for what is broken. They will praise the sophistication and professionalism displayed by the Red Team's approach and results. "Wow," they say, "we have to fix this stuff right now!"

Worse still, it's highly likely that the recommendations from the report and the priorities that follow will be bandaids that cover up, but do not treat the core issues. Objectives will be set to define new detection capabilities so that the specific things the Red Team did this time, can be detected next time. Arbitrary objectives like "improve patch management" for assets within the scope of the findings will become new short-term projects. Strategy will be washed away by the surge of tactical things that must be done. A new mindset will permeate the organization that says "if the Red Team did it, then that's how the bad guys will do it." A timeline will be set, and a new rhythm of meetings and updates will be enabled. It's what happens.

Dose of reality: if the Red Team found it that quickly and was able to exploit it that easily to that end...then you have substantial gaps in your fundamentals that extend well beyond a 3-6 month timeline for resolution. You have a 2+ year journey ahead. It it was that fast and easy, then they didn't take a sophisticated and advanced approach - they simply followed what was readily available using a standard approach and standard set of tools. That means, as you already knew, things are fundamentally broken across the board. You, or those among your organization, probably already know what is broken, and how to fix it. Worse, treating the symptoms of the underlying problems means the underlying problems will persist...to be found once again by another Red Team, or an actual adversary.

Here's my response to the Red Team:

1. No shit.
2. Thank you.
3. Can I get some sympathy? Apathy?
4. Help me...don't just point out what isn't being done. Do something truly constructive.
5. Next time, seek to understand the actual underlying issues creating the gaps, and provide specific advice to address them.

Here's my guidance to the Blue Team:

1. You aren't alone. Drink a pint or two to settle yourself and let it go
2. Try to lift everyone's perspective out of the report to help them see the real systemic issues
3. Map the report's recommendations back to existing initiatives if possible
4. Hold your leadership accountable
5. Bring your own experts if your guidance conflicts with the Red Team

Ok, all kidding aside, Red Team exercises can be really valuable. They can indeed find gaps in your defenses that you thought you had closed. They can bring awareness and perspective to senior leadership if you aren't getting traction or support regarding your security program. They can help you discover things beyond your ability to do so.

Here's my recommendation regarding Red Team exercises.

First of all, wait. If you are a new leader of an organization, and everyone around you is singing the tune that "everything is awesome," then yeah, consider an external assessment by a 3rd party Red Team as a tool to validate and educate. Perhaps your team is simply unaware of what they are supposed to be doing, or lacks the visibility to discover problems that exist. If, however, your team is telling you things are broken, that the program is immature, and if you have a long list of things to fix and your team is making progress, then please, for the sake of your team and your organization, wait. The Red Team exercise, fundamentally will provide no value. It will simply add more things to do among your sea of existing initiatives. Your team knows stuff is broken. Wait until their major initiatives are completed, until your team has done the work they think is important and is comfortable with the results. Wait until you and your team have a level of confidence that you have done your best, have filled the gaps, and are at a state of relative comfort in the maturity of your security program. Wait until you have your fundamentals in place, you are measuring their effectiveness, and you are ensuring processes are executed consistently. Then, by all means, test. But, if you test too early, the results will discourage your staff and can send you down the wrong path. Red Teams have their place and value, it's key that you choose the right time and leverage them in the right way.

I've seen it. Too many times.