Still Managing False Positives?

Still Managing False Positives?

As an Operations Analyst, there are few things more frustrating (aside from not having the resources you need to do your job), than being inundated with inaccurate information. When inaccurate information comes in the form of an alert generated by threat detection technology, we call it a false positive; a standard term used to describe the incorrect identification of something. In fact, the security industry has placed considerable emphasis on the subject of false positives, baking detection fidelity values into product marketing, and basing consulting efforts and operational best practices, on the objective of reducing false positives through technology configurations, rule fidelity measurements, alert “tuning,” and even SIEM rule writing. It is a noble cause, since in the information security world, accuracy of data is paramount.

Within the Security Operations context, I have found that the false positive management discussion usually breaks into two categories; the first being whether or not an individual product is accurately identifying the attribute it claims to (is x really x), and second whether or not the alert generated represents an actionable event that Analysts need to respond to (do I need to act upon x)? In the traditional security team, the Analyst or Incident Handler exists to manage incidents, which themselves represent events which have had a negative impact upon the enterprise in some measurable way. The point is to find bad things, to stop them, to restore things to their safe, working order, and to prevent them from happening again.

I fully subscribed to the urgency and practices of false positive management on individual detection technology (first scenario above) in my early years (circa 2000-2004). I approached detection (and prevention) technology from the perspective of configuring it to only tell me about the most accurate and urgent issues it could find, based on what was relevant to the characteristics of the network the technology was protecting. Before turning the IDS/IPS on, I would spend a considerable amount of time understanding the protected environment, then choosing only the IDS signatures and alerts that best matched the needs of that environment, and preventions for only the most accurate signatures and highest risk events. Even then, the technology by and large, would often inundate Analysts with false positives, and so we would often adjust the signature variables over time to include or exclude certain conditions. This was an effective way to manage the volume of alerts, but if we tuned the IDS too aggressively, we would find the opposite problem; no noise, but also, no detection of things that we did want to know about. We also tested tuning approaches based on the priority of signatures or threats. For example, turning off all informational alerts because they by themselves are rarely actionable anyway. I even recall in the early days, disabling entire network protocols from inspection because we didn’t believe we had them present in our environment; a practice that is clearly dangerous in today’s threat climate. Back then, it was all about managing that upside down triangle we depicted in presentations as the “alert funnel.”

The fundamental issue we were struggling with back then was the isolated perspective of the detection system. We needed to know when certain conditions or payloads were present on the network, however the presence of payload that resembles malicious activity a) isn’t always accurate and b) doesn’t tell us the full story of what actually happened. Even the signatures developed from intelligence that we had 100% confidence in, still produced operational false positives. Case in point, if a web browser is exposed to exploit code served from a website, and the IDS detects a string of characters we told it represents the exploit code, that’s a true positive detection in an isolated context. The session was indeed malicious and the IDS did detect what we told it to. However, what that doesn't tell an Analyst, is whether or not the exploit was successful, and whether or not the endpoint had been affected; namely compromised. There are many additional factors that may determine or prevent impact, and Analysts, generally speaking, only need to manage impactful events. This placed the burden on them to figure out which of those alerts did represent something they needed to act upon; a monotonous task. There is also the variable of network placement of the sensor relative to other mitigating technology. Thus, in an operations context, we found the need to manually pair detection alerts with other logs that would inform us of the outcome of the transaction, and impact observed (if any). As we moved in that direction, we found that it was more advantageous (from a single alert perspective) to shift our focus to and only elevate those alerts that described negative impact, such as C2 detections that were permitted to leave the network perimeter, which themselves have issues with false positives.

The fundamental problem and incorrect practice many operators (including us) made, is that none of these technologies or alerts can tell the full story, and so we should not attempt to derive high fidelity action according to individual alerts from any of them. Event the alerts that try to tell us about impact. Any individual log or alert represents a perspective on an individual portion of an overall event; a possibility, which in the threat context (think kill chain) requires multiple actions or events occurring in sequence over time, involving multiple network and system components to be realized. The only way to automate the optimal elevation of actionable alerts to Analysts is to be able to observe and validate cause and effect in an automated way. That is usually only discernable by looking toward the endpoint, although symptoms of endpoint actions are visible at the network layer (e.g. C2 as evidence of the effect caused by malware).

I’ll give you an example by using web exploit kits (WEK). In a WEK based attack, a user is usually lured to a particular website (often via a phishing email, redirection, malvertisement etc.). There are loads of tools or data feeds out there that try to tell you about this activity or infrastructure used for this purpose. When exposed to the initial page, the WEK will often perform asset identification and profiling so it can determine if it will serve additional content, and what that content or payload will be. This profiling activity is sometimes detectable using network inspection technology. It can even be detected simply based on the presence of URL attributes, and session behavior (based on the static structure of some WEKs). If the browsing asset (or application) matches predefined conditions, exploit code will be served (sometimes from a separate site) to the browsing application. This too can often be detected via network inspection. Next, the exploit code runs on the endpoint itself, within the browsing application, often with the intent of initiating the retrieval of additional content or a binary from a web property, followed by an attempt to automatically execute the retrieved object on the endpoint. At this point, you may have an endpoint application that has been exploited, but the endpoint itself has not been compromised (yet) meaning there is nothing for an Analyst to remediate (except the vulnerability in the application that facilitated the exploit). This too presents opportunities for detection via network inspection AND via endpoint monitoring. At this phase, we have exploit code running on the endpoint, we have a new network session retrieving additional content (think User-Agent strings, URI, request parameters etc.), we have a file download (subject to additional code inspection and sandboxing), and we have a binary waiting to be executed on the endpoint. Lost of places to detect badness. If the binary executes successfully, additional behaviors begin on the endpoint which can be visible via endpoint log and process monitoring, and we will likely have follow-up C2 traffic generated from the malware once installed. Lots of opportunities for detection, and even more opportunities to generate “false positives” in their traditional sense.

So, thinking of the WEK detection paradigm outlined above, where do you want to be notified as an Analyst? What do you want to know? That depends right? Do you want to know that endpoints are being exposed to web based exploits? Probably so. That’s good information to identify risk and root cause. Is that the right time to start action from an investigation and response perspective? Probably not. But what if you don’t have pre-defined content or the ability to detect the other activity that follows the initial exploit action? Should you be alerted at every stage possible just in case something gets missed? You might be inclined to say yes, but that’s where we land into the false positive trap. From the traditional Analyst perspective, the evidence they need to act upon is a sign of compromise, such as when the C2 is generated. That’s the only point at which you know an actionable event that requires remediation has occurred. The other isolated data points simply indicate that something may have happened. Now of course this is a generality that doesn’t always apply, but I think in most cases it does. However, knowing there has been a symptom that reflects possible impact, is not by itself always high fidelity and always actionable. At some point during the IR cycle, you’ll need to discern the cause...which to do, you need evidence collected at the time of the attack...which means you will need some alerts/data about those initial phases of the WEK process after all. However, having the data, doesn’t mean you have to present it all to Analysts.

That’s why thinking of alerts or logs by themselves and expecting some form of actionable fidelity from them doesn’t work. You have to view logs, alerts etc. as meta data. It’s simply an indicator of a possibility.

That’s where we as an industry turned to the world of event correlation and SIEM (circa 2003-2004). We needed  to separate the raw sensor and log data from the Analyst perspective, and we needed to control the Analyst experience independent of native alerts. We found that through correlation, we could treat alerts as meta data actually suppress much of the noise coming from our deployed sensors (at least from the Analyst's perspective), and could elevate only the detections that we could correlate with additional supporting evidence. We could easily suppress alerts while keeping them active for evidence collection. We also found that through correlation, we could monitor for and piece together sequences of events observed from different technologies, and could base our alert decisions once a certain sequence or outcome was reached. It actually does work. We also found that we could fill some of the detection gaps, through further correlation and automated analytics. Effectively, we shifted from alert monitoring to behavior monitoring. Correlating disparate data together to discover a sequence of events, is in effect monitoring for behaviors. This made us hungry for more data, so we turned everything back on (at the log source) and filled our SIEMs with data, and our white boards with behavioral sequences, and our wikis with endless use cases. We applied certain math functions to the data in conjunction with boolean logic, and shazam, we were actually automating the Analyst analytics process, creating highly actionable alerts that truly represented something of interest.

For example, a team I once worked with used correlation to identify and dynamically tag at-risk assets based on threats they were exposed to, then used a separate sets of rules that evaluated those assets within the context of a separate behaviors and alerts. We used the correlation logic to classify data and raw alerts as different indications of behaviors we wanted to compare. In the above WEK example, we may tag an asset exposed to a web exploit kit as at risk of a malware infection. No alerts, just tagging the asset using meta data in the SIEM (ArcSight) in the form of a new correlated event. We were actually producing meta data before it was an industry buzz word. Then, we would monitor those assets that were tagged using a different set of correlation rules that were looking for suspicious behaviors which we had previously defined as evidence of impact. For example, did a host that was tagged as at-risk, also download a file that met our suspicious downloads criteria, and did those hosts generate web activity that matched C2 behavior such as repeated HTTP GET requests to different domains where the post domain URL information was always the same (an example of one behavior we looked for)? Did an at risk host connect to a domain that had not yet been categorized by our URL filters within 1 hour of being exposed (another suspicious behavior)? Did a host that was observed as at-risk, also generate a connection to a known C2 domain within 2 hours of exposure (another behavior)? Logic like that. It actually worked, and worked great. We expanded our imaginations and started searching for net new threat activity that our sensors couldn’t natively identify because they didn’t have the perspective we had, nor the analysis power we used, but we could spot badness based on what behavior those logs represented when joined sequentially and when analyzed with automated functions.

One of my favorite sets of rules we created was based on analysis of over 100 malware samples we analyzed over a period of months. We took 100 unique samples, and from each of them, summarized their delivery and C2 behavioral characteristics. One characteristic that stood out to us was the fact that most samples generated egress web connections that were unusually small in size (smaller than say 100 bytes) and repetitive in some way. We created content we called the “Bytes Out Tracker” that selected certain HTTP activity from endpoints (based on the observations from the malware), counted the total bytes sent per session, and looked for repeats of the same behavior from the same endpoint within the past 1 hour. The result? We found malware infections that nothing else alerted us to, and we found new C2 notes that our intelligence communities were unaware of. Today, we call that behavioral analytics, and industry tells us we need something net new, like big data, machine learning, and AI, to perform that sort of real-time analysis. Maybe so, but we did it nearly a decade ago using the commodity SIEM that our industry loves to hate.

However, as we became better at detecting and thwarting threats, adversaries got better at hiding their activity. That’s when we found the need to expand our detection rule sets (in the native inspection / logging tools) to also identify attributes within network transactions that could indicate malicious intent. That’s different than finding malicious behavior. Cause and effect were becoming more difficult to discern, so we had to look for potential. This expansion of data collection by itself creates a whole new flood of “false positives.” Where before we wanted to know the answer to questions like, “was the host served a malicious file,” now we needed to know things like “did the endpoint download a file that contained signs of obfuscation - regardless of verdict of the code itself,” and “what was the TLD of the domain involved in the web transaction,” or “have any of our hosts ever connected to that domain before?” Questions that were related to looking for suspicious or unusual characteristics vs. overtly malicious functions and behaviors. Rules that indicated sessions contained obfuscation became of high interest to some Analysts, and were high fidelity in some cases. Rules that simply identified file types involved in egress uploads also became very interesting and relevant artifacts. So, we added more sensors and more data to the mix. Suddenly, simply knowing things like the application that generated a network connection became a critical piece of evidence that we analyzed in real-time. In a traditional SOC, if you fed that stream of data to an Analyst, they would go crazy. We moved well beyond that reality, and stopped thinking of individual logs or even sources of logs as actionable data; it was all simply data that we needed to consider.

Overtime, we got good at this practice as well, and could spot anomalies within our data, once again by using commodity SIEM (ArcSight), that critics in industry like to tell you SIEM can’t do. It can, and we did. We also started moving toward industry products, like Fortinet, McAfee NTR, NetWitness, and Palo Alto Networks, that could provide us as much information about network transactions as possible - we just wanted to know what was happening in general so that we could discern intent.

However, that approach also has limitations, is extremely labor intensive, and is extremely expensive. It has a significant dependency upon the SIEM content management team (rule writers) being able to define and code for any possible combination of factors that would represent a true threat sequence. It also depended upon an Analyst team that was comfortable with experimentation, and stopped looking at logs or alerts by themselves as key indicators. It also meant we needed to store and analyze all that data. Now, having said that, an argument could be made that if your team can’t pre-define an attack sequence, they won’t know how to identify it or manually discover it should it happen in the absence of correlation logic. We can only do what we know how to do. The SIEM correlation approach will substantially reduce false positives AND it will also substantially improve your ability to spot things you want to know about...but it doesn’t necessarily help you find the things you don’t know to look for. That’s where intelligence functions began to really grow, because they became essential elements to operational success and the accuracy of threat detection. However, we sort of lost the plot on that.

This is where industry went off the deep end, chasing the rabbit down the down the endless data analysis path; more data, bigger storage, faster systems, more analytics, artificial intelligence, machine learning, different visualization etc. etc. etc. That’s the wrong direction. This is also where boutique services like dark web monitoring, Internet crawling, and related capabilities were born; trying to find the net impact somewhere out there. If you see data for sale on an underground forum, then you know you had a real event, and can move to mitigate issues that will have the most measurable impact to the business. But again, that’s running down the wrong road. It sounds cool, and it is the home of technology innovation, but it doesn’t solve the problem. It’s an insatiable appetite.

Paradigm Shift

Now that I’ve made the case for SIEMs (not meaning to), I want us to abandon that thought process. My point in telling that story is that I hope you can see that it is a never ending battle of trying to make data as valuable as possible in an operational context that is designed to rapidly detect threats in order to rapidly mitigate them and manage impact (possibly prevent damage). Trying to elevate the threats you know about, and find what you don’t know as a means to manage risk isn’t effective. That’s the wrong way to think about threats these days (and it was wrong back then too). Stop thinking about finding the badness that’s lurking invisible in your network or just beyond your field of view. In the words of the rock band Switchfoot, “there’s a new way to be human, something we’ve never been.” You see, we didn’t solve the core issues with our data analytics approach, we’ve been simply seeking the best bandaids to slow the bleeding. You won’t win the detection game. You can’t. It’s reactive in nature and most threats most organizations will face are automated anyway. By the time your multi-million dollar infrastructure finds the highest fidelity thing for your team to respond to and manage, it’s likely that the damage has been done.

You see, it was at the top of my “detection” and analytics game that I was exposed to a different class of Analyst who took a completely different approach. I still remember the event clearly. The conversation wasn’t “how do you best detect x,” or “what products give you the most value,” but rather, “did you know adversary y has updated tool z with x capabilities to go after a, b, and c data sets?” I recall a presentation where an Analyst (by title) presented a timeline of adversary activity that revealed a clear pattern. They weren’t talking about the lower levels of the kill chain, except in passing. Instead they were talking about predicting when and how their adversary would engage them, and with what tools and processes, exploiting what vulnerabilities in which applications. They focused not on finding suspicious behaviors and effects within their environments, and instead focused on defeating the adversary’s ability to operate against their environment to begin with. They studied their adversary, learning their tools, their behaviors, their objectives, their triggers, their operational rhythms, etc. etc. They were students, experts of their adversary. As I watched them in practice, I realized how simple this whole thing really is. You see adversaries operate in the same space we do, using the same technology, with the same constraints, and across the same infrastructure. They have finite capabilities and opportunities. The more you understand how they operate and can defeat their capabilities and remove opportunities, the far less you have to analyze on the back-end to detect and hopefully mitigate their success. Why not try your best to stop or disrupt them as they are acting? Why wait for them to act, hoping that maybe you can catch them based on the trail of blood they left behind? Yeah, I’m talking about prevention, but more than that, I’m talking about actually managing your threat posture relative to your actual adversary. The beauty is, you probably already have the means to do that, and it will cost you far less than you think.

Let’s be honest for a second now. That means, you have to get very real, and very serious about vulnerability management in your environment. Yep, that’s boring and many thrill seeking, people pleasing, and attention grabbing CISOs will have an allergic reaction to that. That means you have to get very intrusive, restrictive, and disruptive with regard to what employees can do with the assets and access they have been given. That means you have to get into the business context of your company, the software development and product release lifecycle, the services establishment and publishing process, IT infrastructure and asset management, risk management etc. You have to take charge of your enterprise security, and you have to do it from the basis of knowing your adversary, knowing yourself, and using that to influence everything that an adversary could touch in your company.

So what does that mean for the SOC Analyst battling the volume of false positives coming from their IDS sensor? It means they need to stop looking at the IDS, and start running reports to discern their risk status, and writing tickets that will help improve the security posture of the enterprise. What assets are externally facing that have exploitable vulnerabilities without mitigating controls in place? Those are your new high fidelity alerts that require action. What endpoints are running old software that needs to be upgraded? Ticket for remediation. What endpoints are outside of configuration compliance? Break-fix. What login portals are missing MFA? Break-fix. What user permissions are allowed on your endpoints? What users have access to applications they don't need? Schedule enterprise wide change control to remove unnecessary access. What network applications are your employees using that create risk? Time to update policy and submit firewall rule change requests to block access. Who is sending data out of the company and where? Time to remind employees of corporate policy, data privacy, and to block access to anything but authorized data and file storage locations. What software is installed on endpoints that could be used by an adversary? Time to remove it. Which firewalls have policies that are not compliant with security standards (i.e. missing detection rules, or allow more than they should)? Time for a configuration audit with remediation requirements. The list goes on and on, but these are the issues the SOC must begin to tackle. The SOC needs to be all about controlling risk in your environment; not passively managing the impact that results from the lack of proper risk management. No need for fancy analytics, no need for machine learning, no need for behavioral analysis of every single employee. No, the real need is to disarm your adversary by closing and locking all your windows and doors, and making yourself as difficult a fortress as possible to break into. Remember, they can't exploit a vulnerability on an application they cannot reach, or for an application that is not vulnerable.

Once you get a handle on that, then it’s time to ensure you have ongoing visibility into the status of those controls, to ensure you do not drift from your optimal posture. That means core success is no longer measured by metrics like by time to respond, but instead success can be defined by the low percentage of exposed vulnerabilities, or uncontrolled entry points, and unauthorized software.

You will also need to instrument capabilities to discover if adversaries are interacting with your environment, based on how you expect them to operate, but this will become a smaller and smaller scope as you mitigate their ability to operate against you in the first place. Why spend millions of dollars on infrastructure, facilities, and Analysts, looking for signs of lateral movement, when you can simply patch the vulnerabilities so the adversary cannot exploit them, control the delivery vectors so the adversary cannot enter, and lock down your endpoints so adversaries cannot use them?

Where do you start? Threat intelligence. I’m not talking about observables or threat indicators or 3rd party feeds (although those are components of an intelligence program), I’m talking about becoming a student of your adversary and truly understanding them. You also need to truly understand your business. Then, look to opportunities to disrupt your adversary; if you can’t stop them, make it as difficult as possible for them. To do that, you are going to need to become very close with your IT department. You may even need to merge your NOC and SOC together, because they will become increasingly dependent upon each other. The tools your NOC needs to manage assets, are the tools your SOC needs to measure risk. The actions your Network and IT staff take, and the ones that the SOC staff need to monitor to ensure they do not add or increase risk. It's time to get the family back together and on mission - the right mission.

False positives? Meh. I don’t think of them anymore.

But the Red Team!

It's happened to me a few times now, and if you operate or have had an audit performed by a third party Red Team, I'm sure it's happened to you. Every time I hear the phrase, "but the Red Team..." I want to...well...respond like this:

It goes something like this: during the normal course of your work in information security, you discover countless programatic and technical deficiencies. You find gaps in patch management, failures to follow defined processes, gaps in access controls and visibility, legacy systems creating risk, assets missing security tools or configured out of compliance, poorly deployed and maintained tools, the list goes on and on. As a practitioner charged with defending this environment and closing the gaps, you have to convey to your leadership a clear definition of the problem, what it will take to solve it, and your progress in doing so. At the same time, you have to be very careful about your message, lest you be viewed as that "sky is falling" security guy who just wants to unplug the Internet. You do your best. Priorities will be selected amongst the sea of problems, some you agree with, some you don't, but you get to work and time passes. Then, incidents happen, priorities shift, staff changes, opinions change, new requests are added etc., further complicating your ability to effectively close the discovered gaps in a measurable and programatic way. Thus progress is slowed, although you are doing a ton of valuable work - the work that is being asked of you and described as your priority. You are making progress on closing the gaps, but all the complicating factors make progress really slow.

Then it happens. In secret, your leadership team launches a Red Team exercise as part of their own due diligence to obtain a 3rd party or somewhat external perspective on things. You may or may not be aware until the final reports are in, and those results will invariably be that the Red Team was able to quickly and easily find a vulnerability, exploit it, gain a foothold, move around the environment, and capture some bright and shiny trophy all without triggering a single alert. They will present their findings to the security organization, pointing out the ease of effort and short time it took for them to capture their flags. They will map their actions to the kill chain to make it look very adversary-like in approach. Big words will be dropped to mask the simplicity of what was actually done. You'll sit there the entire presentation nodding your head in agreement as the results confirm your own observations. "Yep, knew that was an issue." Leadership however, will be shocked, furious, disappointed. They'll ask questions like "why was this so easy?" "why didn't you detect it?" "why didn't you tell me it was this bad?" "didn't you say you already fixed that?" You'll be more shocked by your leader's response than the report itself...stunned into silence, not knowing what to say.

Then, the Red Team leaves (as far as you know), or begins scheduling the next offensive.

Next comes the new set of priorities, and the new mantra on everyone's tongue and measuring stick for every new initiative will be "well the Red Team found..." or "that's not what the Red Team said," or "but that's what the Red Team did." All of the sudden, your expertise doesn't matter. You prior observations, recommendations, and progress are tossed out the window because, "the Red Team..." The junior and inexperienced staff in the organization will look upon the Red Team with awe. They will be heralded as some uber l33t hack team that saved the company from being the victim of not knowing just how bad things were, and celebrated as the savior who finally brought clarity to what really should be done. Everyone will chase the Red Team report, scouring over it to find what must they do to fix the issues found, or to seek blame for what is broken. They will praise the sophistication and professionalism displayed by the Red Team's approach and results. "Wow," they say, "we have to fix this stuff right now!"

Worse still, it's highly likely that the recommendations from the report and the priorities that follow will be bandaids that cover up, but do not treat the core issues. Objectives will be set to define new detection capabilities so that the specific things the Red Team did this time, can be detected next time. Arbitrary objectives like "improve patch management" for assets within the scope of the findings will become new short-term projects. Strategy will be washed away by the surge of tactical things that must be done. A new mindset will permeate the organization that says "if the Red Team did it, then that's how the bad guys will do it." A timeline will be set, and a new rhythm of meetings and updates will be enabled. It's what happens.

Dose of reality: if the Red Team found it that quickly and was able to exploit it that easily to that end...then you have substantial gaps in your fundamentals that extend well beyond a 3-6 month timeline for resolution. You have a 2+ year journey ahead. It it was that fast and easy, then they didn't take a sophisticated and advanced approach - they simply followed what was readily available using a standard approach and standard set of tools. That means, as you already knew, things are fundamentally broken across the board. You, or those among your organization, probably already know what is broken, and how to fix it. Worse, treating the symptoms of the underlying problems means the underlying problems will persist...to be found once again by another Red Team, or an actual adversary.

Here's my response to the Red Team:

1. No shit.
2. Thank you.
3. Can I get some sympathy? Apathy?
4. Help me...don't just point out what isn't being done. Do something truly constructive.
5. Next time, seek to understand the actual underlying issues creating the gaps, and provide specific advice to address them.

Here's my guidance to the Blue Team:

1. You aren't alone. Drink a pint or two to settle yourself and let it go
2. Try to lift everyone's perspective out of the report to help them see the real systemic issues
3. Map the report's recommendations back to existing initiatives if possible
4. Hold your leadership accountable
5. Bring your own experts if your guidance conflicts with the Red Team

Ok, all kidding aside, Red Team exercises can be really valuable. They can indeed find gaps in your defenses that you thought you had closed. They can bring awareness and perspective to senior leadership if you aren't getting traction or support regarding your security program. They can help you discover things beyond your ability to do so.

Here's my recommendation regarding Red Team exercises.

First of all, wait. If you are a new leader of an organization, and everyone around you is singing the tune that "everything is awesome," then yeah, consider an external assessment by a 3rd party Red Team as a tool to validate and educate. Perhaps your team is simply unaware of what they are supposed to be doing, or lacks the visibility to discover problems that exist. If, however, your team is telling you things are broken, that the program is immature, and if you have a long list of things to fix and your team is making progress, then please, for the sake of your team and your organization, wait. The Red Team exercise, fundamentally will provide no value. It will simply add more things to do among your sea of existing initiatives. Your team knows stuff is broken. Wait until their major initiatives are completed, until your team has done the work they think is important and is comfortable with the results. Wait until you and your team have a level of confidence that you have done your best, have filled the gaps, and are at a state of relative comfort in the maturity of your security program. Wait until you have your fundamentals in place, you are measuring their effectiveness, and you are ensuring processes are executed consistently. Then, by all means, test. But, if you test too early, the results will discourage your staff and can send you down the wrong path. Red Teams have their place and value, it's key that you choose the right time and leverage them in the right way.

I've seen it. Too many times.

It's Time to Compete in Information Security

One of the recurring messages I hear from family and friends when I talk to them about information security is, "this is a really big problem, so how do we help people understand, and how do we fix it?" In considering this question, I've also been considering how the general population learns about anything new, and how those new things become trends which later become the standards by which we live. You see, once the consuming public understands something, and demands it, they will tolerate nothing less. I think the answer then is, in product advertising. I think the answer is, it's time providers made information security a competitive topic and advertised it as a competitive advantage over their competition. I think we need to drive consumer understanding by putting the topic directly in front of them in as broad a means as possible. Everyone in our nation consumes, so why not engage everyone in their common practice and force the question upon them: "would you choose the products from provider a or b if you knew one had stronger information security practices than the other? Here's what's at risk if you don't choose wisely."

If you are a consumer who assumes information security is baked into every product and service you purchase, you aren't alone in that assumption, and unfortunately you are gravely wrong. In general, product and services providers, heck, nearly every organization out there except the top few (Fortune x lists), considers information security a nuisance; something they must fund, like car insurance, in order to enter the market and provide goods and services. In fact, I've heard the term "insurance" used countless times by organizations as they describe how they view Information Security. The security program is there just in case. Therefore, InfoSec is generally funded or supported in a minimal way to satisfy regulatory controls or external inquiries, but falls far short of actually addressing security challenges. That places you, the consumer, all of us, at considerable risk. You see when you provide someone your information, whether that be sensitive credit card numbers, or pictures you post online, or information about yourself etc., you are placing your personal security, health, and wealth in that third party's hands. Criminals are out there, constantly trying to access that information so they can sell it for a profit. Identity theft, stolen credit cards, product fraud and other consumer impacts are simply the tip of the iceberg. Worse, some nations are out there stealing that information to drive strategic advantage over other nations in the international trade debate. As we become increasingly dependent on connected systems and data via the Internet, we are placing our way of life and often times our own safety and security into the hands of the providers who connect us with those services. Yet for the most part, those providers view Information Security and Systems Security as a cost of business that must be tightly controlled, else it negatively cut into profit.

There is far more at stake here than you realize. What if your smart phone stops working. The phone works fine, but the interconnectedness of all your apps suddenly stops. Think about that for a minute. Think of all the things you do on a daily basis via your smart phone, tablet, or computer. Think about all the things in your home, work, or school locations that use the Internet or networking in some way. Think about email, what if it's gone? Not just consumer email, but enterprise email as well. What about traffic lights? Grocery store registers? Meters and pumps that control water purification and distribution? What if your bank suddenly freezes all your financial assets because someone impersonating you just transferred a bunch of money to a terrorist organization? What if someone posing as you purchased illegal goods or content via the Internet, and the FBI happened to be monitoring? What if your company suddenly finds themselves up against new competition who keeps delivering exactly the same products and much lower costs to your intended customers? What if your personal emails are suddenly leaked onto the public Internet? What about voice conversations? What if you are battling a medical or personal issue that you haven't informed anyone of, and your records, your medication purchase history, suddenly gets posted to social media? What if someone else is watching your home security camera system or your baby monitor? What if that message you just texted to your friend about how much you hate your boss, suddenly gets forwarded to your boss? What if media outlets stop broadcasting via cable, the radio, or Internet? What if you are in the air and your aircraft's flight control system can no longer connect to GPS information?

We are far more dependent upon technology and interconnectedness that we recognize...and most of those organizations providing the services we are dependent upon view Information Security as a nuisance.

In our world of supply and demand, what consumers demand, providers supply. Providers won't provide what people don't want or aren't willing to pay for. When one provider steps forward and boldly advertises something that catches the attention of the consuming public, it ignites a fire that spreads throughout the industry, sparking competition, and replacing the old with new. Take the iPhone for example. Say what you want to about Apple, but they created a demand that became a new standard of living in the US and globally. However, as I'm sure you can attest, often times as a consumer, you may not be willing to purchase something until your understanding of it changes. Just because something is new and advertised, doesn't mean it will stick. Sometimes, when understanding takes root, it creates a paradigm shift, and everything changes. I recall back in the early 2000s, ordering books online from Amazon.com, back when that's what basically all they sold. I recall friends and family members making fun of me for shopping online rather than in traditional stores. I dared them to try it for themselves. The rest is history. Now Amazon is part of the life of almost every American, and we are no longer satisfied with 10 day shipping times. You see, what you know influences what you do. In fact, I believe that what you know or understand, is reflected in what you will do. It's a cause and effect relationship. Before you understood something, what seemed totally unreasonable or unnecessary to you, becomes a requirement and a new expectation, or a new assumption moving forward, once you understand it. Now, thanks to Amazon Prime, my kids are challenged with learning to wait more than 2 days to receive anything new they ordered or was ordered for them. It's a new standard of living, and a new set of expectations. They, and I, won't tolerate 5-10 business day shipping any longer. I believe that if we can establish common understanding of the security topic and use that understanding to drive consumer demand, then we can really address the problem, because consumers will demand it. The first challenge though, is understanding. You may think you understand the risks because your hear about security in the news, or read about the latest big data breach online. However, I challenge you on that. I think you are aware of some of the impact...but you lack understanding.

Consumers have been inundated with a constant stream of bad "cybersecurity" news from the media over the past decade. The escalating trend of breach after breach, growing larger and larger, has effectively elevated the cybersecurity issue to a mainstream discussion within our nation. We are generally aware, however as I recently wrote about in another post, I believe one of the largest challenges facing us in the information security (InfoSec) discussion today is, we have a broad and increasing sense of awareness on this topic, but a general lack of understanding. What I mean by that is people are generally aware that data and information systems protection is a problem, but they don't understand the depth of the problem, nor what it takes to fully address it. Worse, they assume that what they are aware of is all there is. When you don't fully understand something, you will naturally make a lot of assumptions about it. The media and entertainment industries haven't helped. In fact, I believe they have hindered this understanding. Based on my conversations with people in life, I believe this lack of understanding has led to three dangerous assumptions among consumers; first that those providing products and services they are dependent upon, are naturally doing whatever it takes to address the security problems we face, second that there is simply nothing that can be done about the information security problem and we should just accept things as they are, and third that the bad cybersecurity news is just more background noise from the news media and isn't really a major problem - it's just hype. Regardless of which of these assumptions consumers hold to, this "awareness" causes them to assume that those providers understand the problem and would naturally do whatever it takes to address it, and whatever they provide is sufficient.

However, this lack of understanding also permeates the provider side (product and service organizations). Organizations and corporations recognize that they must do something with regard to protecting sensitive data, including the data their customers provide them. These organizations are driven by producing something that others will consume. That means they look to their consumers to define the requirements for the products or services they will provide. That means guardians of our sensitive data and the systems we're dependent upon for everyday life, are looking to the consumer to establish the requirements for those products and services. When consumers are ignorant, they won't set the right expectations of their providers. When consumers are ignorant, and have a basic level of awareness about the security topic, they will be satisfied with their providers implementing minimal security capabilities that satisfy this basic consumer awareness. This minimally compliant approach falls short of actually fixing the problem.

I'll give you an example. I recently worked for a software company who developed software for the general public. They were what those in the industry call a "business to consumer," or B to C, company. Due to increasing security issues their consumers were experiencing (they became aware of the real problem), they were directly faced with the question, "what should we do, and how far should we go?" They decided to put the question to their customers and developed various examples and simulations that demonstrated various depths of security features that their customers could benefit from. They found that among the sample customers they engaged, the majority actually provided negative feedback when they interacted with the models that had more security features built-in. This translated in business terms to potential loss of customers if they moved to an approach that adopted more security into the product. However, for those customers who were impacted by the low state of security in product, they were willing to tolerate a different, more secure experience, because they now understood the problem. In fact, I recall hearing of customers being surprised by the lack of security in the product, asking, "why didn't you do x already?"

I believe it's time providers started generating a demand for improve information security, by directly advertising and educating their customers on the topic. I think it's time information security became competitive advantage. I think it's time organizations stepped forward and fully exposed the security problem, but demonstrating how they have solved it and their competitors haven't.

The SOC - Why We Get It Wrong

The SOC topic is often controversial, with some championing that SOCs are the ONLY way to go, while some criticizing that SOCs are purely show pieces with no real value relative to the mission they purport to execute. I've live through that controversy all of my professional career. I worked in a SOC for almost 14 years (one of those world-class ones that other organizations try to replicate), and followed that with 2 separate organizations pursuing internal SOC builds for different reasons. During my decade plus in a SOC, I was asked to help "Next-Gen" the SOC a few times; to make significant steps forward in maturity and function. I was also asked to replicate it, both for ourselves and for our customers. I was the guy you talked to when you came to see our SOC or when you engaged our services to either build one yourself, or have us build it for you. I was the guy who you met at our vendor booth at various security conferences, trying to convince you of the values and merits that a SOC (especially an outsource MSSP) has to offer. I've lived it. I found value in it. I think I have an "expert" opinion on it.

So, when I see articles like the one posted recently on Dark Reading that challenged the value of SOCs by exposing that a mere 15% of organizations with SOCs would call them mature, I'm naturally tempted to opine. Even better, when I see people comment to that article or on LinkedIN about that article, using it to justify why SOCs are a waste of time and money, I'm hooked, and I have to say something.

This may surprise you, but I do not advocate that a SOC is for everyone. In fact, I believe that very few organizations in the world would actually truly benefit from what a SOC has to offer; namely a continuous perspective of that organizations threat status, and an effective way to rapidly mobilize and coordinate response actions should they be warranted. Should every organization with a security team build a SOC? In my opinion, no. Should every large security team organize themselves around a SOC concept? Again, in my opinion, no. But even those questions miss the point. People say a SOC is a waste of money and they point to failed SOCs as examples, citing the proof that orgs who tried to build the SOC ended up no more mature than organizations without them. This represents a failure in implementation, and a miss-understanding of what the SOC is. 

The part that is missing from the debate, is that many look to the form of the SOC to define the security organization's function. It's not supposed to be that way. A security organization's function, may be best organized into the form we call a SOC, but starting with the form first fails to recognize the fundamentals necessary to leverage that form.

It's an age old debate: if I want to become a marathon runner, do I start by buying the gear used by the top athletes in the Boston Marathon? No, I use what I have and make incremental progress, maturing my skills until my function necessitates a different form. Dressing like someone, does't help me fit the part. It might help facilitate a mindset shift, and it might help me with motivation. But fundamentally, dressing like a marathon runner does not equip me to run a marathon. Running does. Training does. Diet does. Passion does. Necessity does.

A SOC is no different. It is one expression of how certain organizations have decided to organize and facilitate security operations work to best align their resources within their specific needs and goals. It is the expression of a type of security program, not the goal of all security organizations. This is fundamentally what so many people get wrong in the debate. You don't build a SOC to make your security program mature; the natural maturity of your security program may lead you to building a SOC. Every security organization should strive for operational maturity, and should perform incremental steps at mastering what they do and need to do. The things that are fundamental to a mature security organization will also be fundamental to a SOC, but the two are not synonymous. In fact, some of the key attributes of a "mature SOC" may have no relevance to your actual organization needs at all.

So to sit back and declare SOCs are a waste of time because so many organizations with them have not found maturity by embracing them...completely misses the point. The point is, those organizations were not maturing to begin with. They took a Field of Dreams approach; building it, and hoping the maturity would come. It doesn't work that way.

Is a SOC right for you? That's a difficult question to answer, but I would start by exploring the actual tactile benefits that form would provide for your function, and ask yourself if that form would better enable those functions or not. What are the core challenges you face as a security team? Would organizing your efforts around a SOC solve those challenges? Are the benefits you are chasing solutions to your actual problems, or does the SOC represent a solution to someone else's problems?

My bet is, you don't need a SOC. Fundamentals exist with or without a SOC. Get those right first.