But the Red Team!

It's happened to me a few times now, and if you operate or have had an audit performed by a third party Red Team, I'm sure it's happened to you. Every time I hear the phrase, "but the Red Team..." I want to...well...respond like this:

It goes something like this: during the normal course of your work in information security, you discover countless programatic and technical deficiencies. You find gaps in patch management, failures to follow defined processes, gaps in access controls and visibility, legacy systems creating risk, assets missing security tools or configured out of compliance, poorly deployed and maintained tools, the list goes on and on. As a practitioner charged with defending this environment and closing the gaps, you have to convey to your leadership a clear definition of the problem, what it will take to solve it, and your progress in doing so. At the same time, you have to be very careful about your message, lest you be viewed as that "sky is falling" security guy who just wants to unplug the Internet. You do your best. Priorities will be selected amongst the sea of problems, some you agree with, some you don't, but you get to work and time passes. Then, incidents happen, priorities shift, staff changes, opinions change, new requests are added etc., further complicating your ability to effectively close the discovered gaps in a measurable and programatic way. Thus progress is slowed, although you are doing a ton of valuable work - the work that is being asked of you and described as your priority. You are making progress on closing the gaps, but all the complicating factors make progress really slow.

Then it happens. In secret, your leadership team launches a Red Team exercise as part of their own due diligence to obtain a 3rd party or somewhat external perspective on things. You may or may not be aware until the final reports are in, and those results will invariably be that the Red Team was able to quickly and easily find a vulnerability, exploit it, gain a foothold, move around the environment, and capture some bright and shiny trophy all without triggering a single alert. They will present their findings to the security organization, pointing out the ease of effort and short time it took for them to capture their flags. They will map their actions to the kill chain to make it look very adversary-like in approach. Big words will be dropped to mask the simplicity of what was actually done. You'll sit there the entire presentation nodding your head in agreement as the results confirm your own observations. "Yep, knew that was an issue." Leadership however, will be shocked, furious, disappointed. They'll ask questions like "why was this so easy?" "why didn't you detect it?" "why didn't you tell me it was this bad?" "didn't you say you already fixed that?" You'll be more shocked by your leader's response than the report itself...stunned into silence, not knowing what to say.

Then, the Red Team leaves (as far as you know), or begins scheduling the next offensive.

Next comes the new set of priorities, and the new mantra on everyone's tongue and measuring stick for every new initiative will be "well the Red Team found..." or "that's not what the Red Team said," or "but that's what the Red Team did." All of the sudden, your expertise doesn't matter. You prior observations, recommendations, and progress are tossed out the window because, "the Red Team..." The junior and inexperienced staff in the organization will look upon the Red Team with awe. They will be heralded as some uber l33t hack team that saved the company from being the victim of not knowing just how bad things were, and celebrated as the savior who finally brought clarity to what really should be done. Everyone will chase the Red Team report, scouring over it to find what must they do to fix the issues found, or to seek blame for what is broken. They will praise the sophistication and professionalism displayed by the Red Team's approach and results. "Wow," they say, "we have to fix this stuff right now!"

Worse still, it's highly likely that the recommendations from the report and the priorities that follow will be bandaids that cover up, but do not treat the core issues. Objectives will be set to define new detection capabilities so that the specific things the Red Team did this time, can be detected next time. Arbitrary objectives like "improve patch management" for assets within the scope of the findings will become new short-term projects. Strategy will be washed away by the surge of tactical things that must be done. A new mindset will permeate the organization that says "if the Red Team did it, then that's how the bad guys will do it." A timeline will be set, and a new rhythm of meetings and updates will be enabled. It's what happens.

Dose of reality: if the Red Team found it that quickly and was able to exploit it that easily to that end...then you have substantial gaps in your fundamentals that extend well beyond a 3-6 month timeline for resolution. You have a 2+ year journey ahead. It it was that fast and easy, then they didn't take a sophisticated and advanced approach - they simply followed what was readily available using a standard approach and standard set of tools. That means, as you already knew, things are fundamentally broken across the board. You, or those among your organization, probably already know what is broken, and how to fix it. Worse, treating the symptoms of the underlying problems means the underlying problems will persist...to be found once again by another Red Team, or an actual adversary.

Here's my response to the Red Team:

1. No shit.
2. Thank you.
3. Can I get some sympathy? Apathy?
4. Help me...don't just point out what isn't being done. Do something truly constructive.
5. Next time, seek to understand the actual underlying issues creating the gaps, and provide specific advice to address them.

Here's my guidance to the Blue Team:

1. You aren't alone. Drink a pint or two to settle yourself and let it go
2. Try to lift everyone's perspective out of the report to help them see the real systemic issues
3. Map the report's recommendations back to existing initiatives if possible
4. Hold your leadership accountable
5. Bring your own experts if your guidance conflicts with the Red Team

Ok, all kidding aside, Red Team exercises can be really valuable. They can indeed find gaps in your defenses that you thought you had closed. They can bring awareness and perspective to senior leadership if you aren't getting traction or support regarding your security program. They can help you discover things beyond your ability to do so.

Here's my recommendation regarding Red Team exercises.

First of all, wait. If you are a new leader of an organization, and everyone around you is singing the tune that "everything is awesome," then yeah, consider an external assessment by a 3rd party Red Team as a tool to validate and educate. Perhaps your team is simply unaware of what they are supposed to be doing, or lacks the visibility to discover problems that exist. If, however, your team is telling you things are broken, that the program is immature, and if you have a long list of things to fix and your team is making progress, then please, for the sake of your team and your organization, wait. The Red Team exercise, fundamentally will provide no value. It will simply add more things to do among your sea of existing initiatives. Your team knows stuff is broken. Wait until their major initiatives are completed, until your team has done the work they think is important and is comfortable with the results. Wait until you and your team have a level of confidence that you have done your best, have filled the gaps, and are at a state of relative comfort in the maturity of your security program. Wait until you have your fundamentals in place, you are measuring their effectiveness, and you are ensuring processes are executed consistently. Then, by all means, test. But, if you test too early, the results will discourage your staff and can send you down the wrong path. Red Teams have their place and value, it's key that you choose the right time and leverage them in the right way.

I've seen it. Too many times.

It's Time to Compete in Information Security

One of the recurring messages I hear from family and friends when I talk to them about information security is, "this is a really big problem, so how do we help people understand, and how do we fix it?" In considering this question, I've also been considering how the general population learns about anything new, and how those new things become trends which later become the standards by which we live. You see, once the consuming public understands something, and demands it, they will tolerate nothing less. I think the answer then is, in product advertising. I think the answer is, it's time providers made information security a competitive topic and advertised it as a competitive advantage over their competition. I think we need to drive consumer understanding by putting the topic directly in front of them in as broad a means as possible. Everyone in our nation consumes, so why not engage everyone in their common practice and force the question upon them: "would you choose the products from provider a or b if you knew one had stronger information security practices than the other? Here's what's at risk if you don't choose wisely."

If you are a consumer who assumes information security is baked into every product and service you purchase, you aren't alone in that assumption, and unfortunately you are gravely wrong. In general, product and services providers, heck, nearly every organization out there except the top few (Fortune x lists), considers information security a nuisance; something they must fund, like car insurance, in order to enter the market and provide goods and services. In fact, I've heard the term "insurance" used countless times by organizations as they describe how they view Information Security. The security program is there just in case. Therefore, InfoSec is generally funded or supported in a minimal way to satisfy regulatory controls or external inquiries, but falls far short of actually addressing security challenges. That places you, the consumer, all of us, at considerable risk. You see when you provide someone your information, whether that be sensitive credit card numbers, or pictures you post online, or information about yourself etc., you are placing your personal security, health, and wealth in that third party's hands. Criminals are out there, constantly trying to access that information so they can sell it for a profit. Identity theft, stolen credit cards, product fraud and other consumer impacts are simply the tip of the iceberg. Worse, some nations are out there stealing that information to drive strategic advantage over other nations in the international trade debate. As we become increasingly dependent on connected systems and data via the Internet, we are placing our way of life and often times our own safety and security into the hands of the providers who connect us with those services. Yet for the most part, those providers view Information Security and Systems Security as a cost of business that must be tightly controlled, else it negatively cut into profit.

There is far more at stake here than you realize. What if your smart phone stops working. The phone works fine, but the interconnectedness of all your apps suddenly stops. Think about that for a minute. Think of all the things you do on a daily basis via your smart phone, tablet, or computer. Think about all the things in your home, work, or school locations that use the Internet or networking in some way. Think about email, what if it's gone? Not just consumer email, but enterprise email as well. What about traffic lights? Grocery store registers? Meters and pumps that control water purification and distribution? What if your bank suddenly freezes all your financial assets because someone impersonating you just transferred a bunch of money to a terrorist organization? What if someone posing as you purchased illegal goods or content via the Internet, and the FBI happened to be monitoring? What if your company suddenly finds themselves up against new competition who keeps delivering exactly the same products and much lower costs to your intended customers? What if your personal emails are suddenly leaked onto the public Internet? What about voice conversations? What if you are battling a medical or personal issue that you haven't informed anyone of, and your records, your medication purchase history, suddenly gets posted to social media? What if someone else is watching your home security camera system or your baby monitor? What if that message you just texted to your friend about how much you hate your boss, suddenly gets forwarded to your boss? What if media outlets stop broadcasting via cable, the radio, or Internet? What if you are in the air and your aircraft's flight control system can no longer connect to GPS information?

We are far more dependent upon technology and interconnectedness that we recognize...and most of those organizations providing the services we are dependent upon view Information Security as a nuisance.

In our world of supply and demand, what consumers demand, providers supply. Providers won't provide what people don't want or aren't willing to pay for. When one provider steps forward and boldly advertises something that catches the attention of the consuming public, it ignites a fire that spreads throughout the industry, sparking competition, and replacing the old with new. Take the iPhone for example. Say what you want to about Apple, but they created a demand that became a new standard of living in the US and globally. However, as I'm sure you can attest, often times as a consumer, you may not be willing to purchase something until your understanding of it changes. Just because something is new and advertised, doesn't mean it will stick. Sometimes, when understanding takes root, it creates a paradigm shift, and everything changes. I recall back in the early 2000s, ordering books online from Amazon.com, back when that's what basically all they sold. I recall friends and family members making fun of me for shopping online rather than in traditional stores. I dared them to try it for themselves. The rest is history. Now Amazon is part of the life of almost every American, and we are no longer satisfied with 10 day shipping times. You see, what you know influences what you do. In fact, I believe that what you know or understand, is reflected in what you will do. It's a cause and effect relationship. Before you understood something, what seemed totally unreasonable or unnecessary to you, becomes a requirement and a new expectation, or a new assumption moving forward, once you understand it. Now, thanks to Amazon Prime, my kids are challenged with learning to wait more than 2 days to receive anything new they ordered or was ordered for them. It's a new standard of living, and a new set of expectations. They, and I, won't tolerate 5-10 business day shipping any longer. I believe that if we can establish common understanding of the security topic and use that understanding to drive consumer demand, then we can really address the problem, because consumers will demand it. The first challenge though, is understanding. You may think you understand the risks because your hear about security in the news, or read about the latest big data breach online. However, I challenge you on that. I think you are aware of some of the impact...but you lack understanding.

Consumers have been inundated with a constant stream of bad "cybersecurity" news from the media over the past decade. The escalating trend of breach after breach, growing larger and larger, has effectively elevated the cybersecurity issue to a mainstream discussion within our nation. We are generally aware, however as I recently wrote about in another post, I believe one of the largest challenges facing us in the information security (InfoSec) discussion today is, we have a broad and increasing sense of awareness on this topic, but a general lack of understanding. What I mean by that is people are generally aware that data and information systems protection is a problem, but they don't understand the depth of the problem, nor what it takes to fully address it. Worse, they assume that what they are aware of is all there is. When you don't fully understand something, you will naturally make a lot of assumptions about it. The media and entertainment industries haven't helped. In fact, I believe they have hindered this understanding. Based on my conversations with people in life, I believe this lack of understanding has led to three dangerous assumptions among consumers; first that those providing products and services they are dependent upon, are naturally doing whatever it takes to address the security problems we face, second that there is simply nothing that can be done about the information security problem and we should just accept things as they are, and third that the bad cybersecurity news is just more background noise from the news media and isn't really a major problem - it's just hype. Regardless of which of these assumptions consumers hold to, this "awareness" causes them to assume that those providers understand the problem and would naturally do whatever it takes to address it, and whatever they provide is sufficient.

However, this lack of understanding also permeates the provider side (product and service organizations). Organizations and corporations recognize that they must do something with regard to protecting sensitive data, including the data their customers provide them. These organizations are driven by producing something that others will consume. That means they look to their consumers to define the requirements for the products or services they will provide. That means guardians of our sensitive data and the systems we're dependent upon for everyday life, are looking to the consumer to establish the requirements for those products and services. When consumers are ignorant, they won't set the right expectations of their providers. When consumers are ignorant, and have a basic level of awareness about the security topic, they will be satisfied with their providers implementing minimal security capabilities that satisfy this basic consumer awareness. This minimally compliant approach falls short of actually fixing the problem.

I'll give you an example. I recently worked for a software company who developed software for the general public. They were what those in the industry call a "business to consumer," or B to C, company. Due to increasing security issues their consumers were experiencing (they became aware of the real problem), they were directly faced with the question, "what should we do, and how far should we go?" They decided to put the question to their customers and developed various examples and simulations that demonstrated various depths of security features that their customers could benefit from. They found that among the sample customers they engaged, the majority actually provided negative feedback when they interacted with the models that had more security features built-in. This translated in business terms to potential loss of customers if they moved to an approach that adopted more security into the product. However, for those customers who were impacted by the low state of security in product, they were willing to tolerate a different, more secure experience, because they now understood the problem. In fact, I recall hearing of customers being surprised by the lack of security in the product, asking, "why didn't you do x already?"

I believe it's time providers started generating a demand for improve information security, by directly advertising and educating their customers on the topic. I think it's time information security became competitive advantage. I think it's time organizations stepped forward and fully exposed the security problem, but demonstrating how they have solved it and their competitors haven't.

The SOC - Why We Get It Wrong

The SOC topic is often controversial, with some championing that SOCs are the ONLY way to go, while some criticizing that SOCs are purely show pieces with no real value relative to the mission they purport to execute. I've live through that controversy all of my professional career. I worked in a SOC for almost 14 years (one of those world-class ones that other organizations try to replicate), and followed that with 2 separate organizations pursuing internal SOC builds for different reasons. During my decade plus in a SOC, I was asked to help "Next-Gen" the SOC a few times; to make significant steps forward in maturity and function. I was also asked to replicate it, both for ourselves and for our customers. I was the guy you talked to when you came to see our SOC or when you engaged our services to either build one yourself, or have us build it for you. I was the guy who you met at our vendor booth at various security conferences, trying to convince you of the values and merits that a SOC (especially an outsource MSSP) has to offer. I've lived it. I found value in it. I think I have an "expert" opinion on it.

So, when I see articles like the one posted recently on Dark Reading that challenged the value of SOCs by exposing that a mere 15% of organizations with SOCs would call them mature, I'm naturally tempted to opine. Even better, when I see people comment to that article or on LinkedIN about that article, using it to justify why SOCs are a waste of time and money, I'm hooked, and I have to say something.

This may surprise you, but I do not advocate that a SOC is for everyone. In fact, I believe that very few organizations in the world would actually truly benefit from what a SOC has to offer; namely a continuous perspective of that organizations threat status, and an effective way to rapidly mobilize and coordinate response actions should they be warranted. Should every organization with a security team build a SOC? In my opinion, no. Should every large security team organize themselves around a SOC concept? Again, in my opinion, no. But even those questions miss the point. People say a SOC is a waste of money and they point to failed SOCs as examples, citing the proof that orgs who tried to build the SOC ended up no more mature than organizations without them. This represents a failure in implementation, and a miss-understanding of what the SOC is. 

The part that is missing from the debate, is that many look to the form of the SOC to define the security organization's function. It's not supposed to be that way. A security organization's function, may be best organized into the form we call a SOC, but starting with the form first fails to recognize the fundamentals necessary to leverage that form.

It's an age old debate: if I want to become a marathon runner, do I start by buying the gear used by the top athletes in the Boston Marathon? No, I use what I have and make incremental progress, maturing my skills until my function necessitates a different form. Dressing like someone, does't help me fit the part. It might help facilitate a mindset shift, and it might help me with motivation. But fundamentally, dressing like a marathon runner does not equip me to run a marathon. Running does. Training does. Diet does. Passion does. Necessity does.

A SOC is no different. It is one expression of how certain organizations have decided to organize and facilitate security operations work to best align their resources within their specific needs and goals. It is the expression of a type of security program, not the goal of all security organizations. This is fundamentally what so many people get wrong in the debate. You don't build a SOC to make your security program mature; the natural maturity of your security program may lead you to building a SOC. Every security organization should strive for operational maturity, and should perform incremental steps at mastering what they do and need to do. The things that are fundamental to a mature security organization will also be fundamental to a SOC, but the two are not synonymous. In fact, some of the key attributes of a "mature SOC" may have no relevance to your actual organization needs at all.

So to sit back and declare SOCs are a waste of time because so many organizations with them have not found maturity by embracing them...completely misses the point. The point is, those organizations were not maturing to begin with. They took a Field of Dreams approach; building it, and hoping the maturity would come. It doesn't work that way.

Is a SOC right for you? That's a difficult question to answer, but I would start by exploring the actual tactile benefits that form would provide for your function, and ask yourself if that form would better enable those functions or not. What are the core challenges you face as a security team? Would organizing your efforts around a SOC solve those challenges? Are the benefits you are chasing solutions to your actual problems, or does the SOC represent a solution to someone else's problems?

My bet is, you don't need a SOC. Fundamentals exist with or without a SOC. Get those right first.