Groundhog Day in the Application Security World

February 1, 2012

By Michael Pearce, a Security Consultant and Researcher at Neohapsis

Throughout the US on Groundhog Day, an inordinate amount of media attention will be given to small furry creatures and whether or not they emerge into bright sunlight or cloudy skies. In a tradition that may seem rather topsy-turvy to those not familiar with it, the story says that if the groundhog sees his shadow (indicating the sun is shining), he returns to his hole to sleep for six more weeks and avoid the winter weather that is to come.

Similarly, when a company comes into the world of security and begins to endure the glare of security testing, the shadow of what they find can be enough to send them back into hiding. However, with the right preparation and mindset, businesses can not only withstand the sight of insecurity, they can begin to make meaningful and incremental improvements to ensure that the next time they face the sun the shadow is far less intimidating.

Hundreds or thousands of issues – Why?

It is not uncommon for a Neohapsis consultant to find hundreds of potential issues to sort through when assessing a legacy application or website for the first time. This can be due to a number of reasons, but the most prominent are:

  1. Security tools that are paranoid/badly tuned/misunderstood
  2. Lack of developer security awareness
  3. Threats and technologies have evolved since the application was designed/deployed/developed

Security Tools that are Paranoid/Badly Tuned/Misunderstood

Security testing and auditing tools, by their nature, have to be flexible and able to work in most environments and at various levels of paranoia. Because of this, if they are not configured and interpreted with the specifics of your application in mind they will often find a large number of issues, of which the majority are noise that should be ignored until the more important issues are fixed. If you have a serious, unauthenticated, SQL injection that exposes plain-text credit card and payment details, you probably shouldn’t a moment’s thought stressing about whether your website allows 4 or 5 failed logins before locking an account.

Lack of Developer Security Awareness

Developers are human (at least in my experience!), and have all the usual foibles of humanity. They are affected by business pressures to release first and fix bugs later, with the result that security bugs may be de-prioritized down as “no-one will find that” and so “later” never comes. Developers also are often taught about security as an addition rather than a core concept. For instance, when I was learning programming, I was first taught to construct SQL strings and verbatim webpage output and only much later to use parameterized queries and HTML encoding. As a result, even though I know better, I sometimes find myself falling into bad practices that could introduce SQL injection or cross-site scripting, as the practices that introduce these threats come more naturally to me than the secure equivalents.

Threats and Technologies have Evolved Since the Application was Designed/Deployed/Developed

To make it even harder to manage security, many legacy applications are developed in old technologies which are either unaware of security issues, have no way of dealing with them, or both. For instance, while SQL injection has been known about for around 15 years, and cross-site scripting a little less than that, some are far more recent, such as clickjacking and CSS history stealing.

When an application was developed without awareness of a threat, it is often more vulnerable to it, and when it was built on a technology that was less mature in approaching the threat remediating the issues can be far more difficult. For instance, try remediating SQL injection in a legacy ASP application by changing queries from string concatenation to parameterized queries (ADODB objects aren’t exactly elegant to use!).

Dealing with issues

Once you have found issues, then comes the daunting task of prioritizing, managing, and preventing their reoccurrence. This is the part that can bring the shock, and the part that can require the most care, as this is a task in managing complexity.

The response to issues requires not only looking at what you have found previously, but also what you have to do, and where you want to go. Breaking this down:

  1. Understand the Past – Deal with existing issues
  2. Manage the Present – Remedy old issues, prevent introduction of new issues where possible
  3.  Prepare for the Future – Expect new threats to arise

Understand the Past – Deal with Existing Issues

When dealing with security reports, it is important to always be psychologically and organizationally prepared for what you find. As already discussed, this is often unpleasant and the first reactions can lead to dangerous behaviors such as overreaction (“fire the person responsible”) or disillusionment (“we couldn’t possibly fix all that!”). The initial results may be frightening, but flight is not an option, so you need to fight.

To understand what you have in front of you, and to react appropriately, it is imperative that the person interpreting the results understands the tools used to develop the application; the threats surrounding the application; and the security tool and its results. If your organization is not confident in this ability, consider getting outside help or consultants (such as Neohapsis) in to explain the background and context of your findings.

 Manage the present – Remedy old issues, prevent introduction of new issues where possible

Much like any software bug or defect, once you have an idea of what your overall results mean you should start making sense of them. This can be greatly aided through the use of a system (such as Neohapsis Security Manager) which can take vulnerability data from a large number of sources and track issues across time in a similar way to a bug tracker.

Issues found should then be dealt with in order of the threat they present to your application and organization. We have often observed a tendency to go for the vulnerabilities labeled as “critical” by a tool, irrespective of their meaning in the context of your business and application. A SQL injection bug in your administration interface that is only accessible by trusted users is probably a lot less serious than a logic flaw that allows users to order items and modify the price communicated and charged to zero.

Also, if required, your organization should rapidly institute training and awareness programs so that no more avoidable issues are introduced. This can be aided by integrating security testing into your QA and pre-production testing.

 Prepare for the future – Expect new threats to arise

Nevertheless, even if you do everything right, and even if your developers do not introduce any avoidable vulnerabilities, new issues will probably be found as the threats evolve. To detect these, you need to regularly have security tests performed (both human and automated), keep up with the security state of the technologies in use, and have plans in place to deal with any new issues that are found.

Closing

It is not unusual to find a frightening degree of insecurity when you first bring your applications into the world of security testing, but diving back to hide is not prudent. Utilizing the right experience and tools can turn being afraid of your own shadow into being prepared for the changes to come. After all, if the cloud isn’t on the horizon for your company then you are probably already immersed in it.


Articulating the Value of Security (or, Security is not the point)

November 15, 2011

It’s an uphill battle to convince the decision-makers in any business that they need to invest in security. Why? Because deep down, most people think security is an annoying layer of cost and inconvenience.  If you walk in and tell them, “We need more security,” they hear, “We need a more annoying layer of cost and inconvenience.”

Getting Buy-In

Getting executive buy-in for security products and services today means understanding what drives your company’s security purchase decisions. Fear, uncertainty and doubt are not the cleverest tools to use anymore. Businesses want something that sometimes seems like a foreign concept to the security profession: value.  If you don’t adapt and start answering the questions your business is really interested in, you’ll never get the green light on new projects and upgrades.  Remember, nobody wants security; they want the benefits of security. Your family members don’t want the finest deadbolt on the front door because of the excellence of its engineering or its impact resistance. They want a comfortable, happy place to live.

Achieve Objectives

Businesses also want something other than security. If a bank manager has a mandate to reduce expenses related to bank tellers, she has a couple of options. She could fire all the tellers and lock up all the bank branches, but then the bank would have no interface with its customers. Or she could take all the money, put it in piles on the street corner under a clipboard that says, “Take what you want, but write it down so we may balance your account.” That wouldn’t work either.  The best solution for reducing teller expenses is to take the money, put in on the street corner locked in a box with a computer attached, and give customers a low-cost plastic card for authentication and auditing.  Security was never the point of creating the automated teller machine. The bank had a business objective and achieved it by using some security.

A Tool in Your Toolbox

That is precisely how we all should think of security: as a way of helping companies achieve the goals or value they seek.  Business managers, especially executives at the highest levels of an organization, have a very simple, indirect view of security. They don’t think of it as security, exactly. They think of it as a tool in the corporate toolbox for enabling business. For example, the manager responsible for a critical business application wants a few things: He wants to know who is using his website; he wants to ensure that everyone can do everything on that site they need to do; he has a lot of users doing a lot of things, so he needs an easy way to manage it; and at the end of the day or the end of the quarter, he needs a report telling him what has happened so that he can improve customer satisfaction, reduce errors and increase profits.  In that example we have all four fundamental categories of security—authentication, authorization, administration and audit—but the manager doesn’t think of security once! That’s because security is not the point.

Focus on Value 

Whenever possible, security professionals should purge the word “security” from their vocabulary. Instead, answer the questions inside your bossyou’re your customer’s head, and don’t simply spout the ways security keeps bad things from happening.  Upper management thinks in terms of money, not security. What people will be needed? What headcount can we reduce? How much will it cost? How much will we save? What new revenue can we earn as a result of this investment? And they think not in terms of security risks, but in terms of credit risk, market risks and operational risks. That’s where security professionals can shine.  For any business problem, you should be prepared to help your management identify the ways that the authentication, authorization, administration or audit solutions you’re proposing will solve their problem or help customers.  Remember, it is not our job to secure the network. It’s our job to secure the business.

- Steve Hunt


Service Provider Scoping Angst

July 15, 2011

By Patrick Harbauer

Over the past several months we have had many service providers come to us wringing their hands wondering if they should go through the ROC process. They may offer cloud services or a managed security service, for example, and they keep running up against QSA’s peppering them with questions when the QSA is assessing one of the service provider’s customers. Or the sales cycle repeatedly hits road blocks when a potential customer’s InfoSec department raises a red flag and wants to know what security controls the service provider has in place to protect their customers from a security breach.

I equate this dilemma that service providers face to going to the hardware store to buy an extension cord. If the hardware store carries extension cords from four manufactures, three of the manufactures have “UL Listed” on their packaging and one has “Not UL Listed” on their packaging, would anyone buy the “Not UL Listed” extension cord? Absolutely not!

So, if the service provider decides to engage us to pursue PCI compliance, they begin to wring their hands again when trying to determine what is in scope. Client employees, and sometimes even the individuals who hired us, ask “Why are we going through this again? We don’t process, store or transmit payment card information?!?!?!??”

Again, I turn to the UL analogy. The extension cord manufacturer may wonder if they should certify the male end of the extension cord, the female end or just the wire.” They could spin their wheels for a long time trying to determine what the scope should be. This is where service providers need to leverage their QSA’s expertise. Again, using my extension cord analogy (OK, so maybe it’s a little goofy but it works for me!!) UL will most likely say, “Mr. Customer, if we are going to put our logo on this product, we have to use our best judgment and examine all aspects of this extension cord to make sure that it is safe to use and won’t cause a fire that results in one or more deaths. If you are not willing to let us perform what, in our best judgment, is a thorough set of testing procedures on any aspect of your product that we deem necessary, we cannot certify your product.”

As QSA’s assessing a service provider’s products and services for PCI compliance, we must be allowed to use our best judgment to determine what components of a service provider’s environment are in scope and the PCI requirements and testing procedures that are applicable. It is our responsibility as QSA’s to make sure that a merchant doesn’t “burn their business” as the result of making the service provider’s solution a part of their CDE only to find out that the solution is not properly secured in the spirit of the PCI DSS.


What Security Teams Can Learn From The Department of Streets & Sanitation

February 1, 2011

As I sit here typing in beautiful Chicago, a massive blizzard is just starting to hit outside.  This storm is expected to drop between 12 to 20 inches over the next 24 hours, which could be the most snowfall in over 40 years.  Yet in spite of the weather, the citizens and city workers remain calm, confident in the fact that they know how to handle an incident like this.  A joke has been going around on twitter about the weather today – “Other cities call this a snowpacolypse.  Chicago calls it ‘Tuesday’”.

Northern cities like Chicago have been dealing with snowstorms and snow management for decades, and have gotten pretty stable at it.  Yet, when cities fail at it, there can be dramatic consequences – in 1979, the incumbent mayor of Chicago, Michael Bilandic lost to a challenger, with his poor response to a blizzard cited as one of the main reasons for his defeat.

The same crisis management practices, as well as the same negative consequences attached to failure, apply to information security organizations today.  Security teams should pay attention to what their better established, less glamorous counterparts in the Department of Streets and Sanitation do to handle a crisis like two feet of snow.

So, how do cities adequately prepare for blizzards?  The preparations can be summarized into four main points:

  • Prepare the cleanup plan in advance
  • Get as much early warning as possible
  • Communicate with the public about how to best protect themselves
  • Handle the incident as it unfolds to reduce loss of continuity

These steps are all also hallmarks of a mature security program:

  • Have an incident response plan in place in advance of a crisis
  • Utilize real-time detection methods to understand the situation
  • Ensure end users are aware of the role that they play in information security
  • Remediate security incidents with as little impact to the business as possible

An information security program that is proactive in preparing for security incidents is one that is going to be the most successful in the long run.  It will gain the appreciation of both end users and business owners, by reducing the overall impact of a security event.  Because as winter in Chicago shows us – you can’t stop a snowstorm from coming.  Security incidents are going to happen.  The test of a truly mature organization is how they react to the problem once it arrives.


PCI Surprises

January 31, 2011

By Patrick Harbauer

Whenever we perform a PCI assessment for a new client, we invariably have the Gomer Pyle “Surprise!, surprise!” conversation with IT management. And the outcome of the conversation is that IT security controls are more closely monitored and the overall security posture of the organization improves. Here are a few examples:

Swiss Cheese Firewalls – When we perform a PCI assessment, we take firewall rule set reviews very seriously. Besides finding the obvious “ANY ANY” rule infractions, we find rules that were meant to be “temporary for testing” or rules that are no longer needed because entire environments have been decommissioned. It isn’t uncommon to see 10-20% of the firewall rules removed or tightened to allow only protocols, IP’s and ports that are actually required for the cardholder environment to function properly.

Missing Patches – Time and again we find in-scope systems that are not properly patched. This is usually due to overtaxed IT staff who don’t find the time to patch systems or a malfunctioning patching software solution. And in some cases administrators have been burned by a patch that brought down an application and vow to never patch again. What we usually find is that “practice makes perfect” with patching. Organizations that are up to date on patches have well-defined processes and document procedures to perform patching. And that leads us to our next issue…

One Environment – In many cases, organizations that are not up-to-date with their patches do not have isolated test/development and production environments. Besides being a PCI violation to have test/development and production systems on the same network and/or servers, if you do not have a test environment that mirrors production, you are more likely to break production applications when you patch. You will be much more successful remaining current with patches if you have a test environment that mirrors production and where you can address issues before applying the patches to production systems.

These are just a few examples of what we see when performing PCI assessments for new clients and illustrates some of the benefits that come out of a PCI assessment.


The value of life and acceptable risk

December 23, 2010

Is it ever okay to accept the loss of life as an acceptable risk to doing business?

First off, is this even reasonable? I believe it is. Though not the best approach to calculating the cost vs benefit of a given security measure, it can be enlightening to look at past and present choices and see what they indicate about the value placed on life by how much money was spent trying to protect it.

But life is invaluable…

By value, I don’t simply mean money. Thou for simplicity sake I will use it for the rest of this post. Most people would say that life is invaluable. But that notion, though admirable and on some level true, it is not an accurate statement. Some people would trade theirs or someone elses life for cash. While others would trade their own life for another. In both cases they have, potentially unknowingly, attached a value to life…or at least a particular persons life.

It’s either valued or it’s not…

By definition, if something does not have a defined value it is either worth nothing or everything. Given that society won’t allow life to be valued at nothing (at least usually) then without a defined value, life is invaluable…or in other words…when placed in your hands, there is zero tolerance for failure to protect.

So? It’s always job number 1…

If you’ve been in the security industry for any measurable time, you will recognize the following priority list from somewhere. It shows up as the default when I do Incident Response program development.

1. Preserve life
2. Prevent physical damage to personnel, facilities, or systems
3. Prevent financial loss
…etc…

This presumes that life is the most valuable item a business/government/entity must protect. Very few, if any security professionals will face the protection of life as priority one in their career. Typically we give lip service to the priority of life since that firewall we just bought doesn’t protect life…at least directly.  But what happens when your entire reason for existing is making something safer?

Safer vs Zero tolerance

I chose the term “safer” very carefully and on purpose. It means some risk, or level of failure, is acceptable and still be considered a success. If your entire reason to exist is to protect life, you can not calculate the value of the security measure unless you know the value of life. As has been shown time and time again through history…nothing can be made perfectly safe…ala zero tolerance. But when tasked with zero tolerance, or zero breaches, and you have no understanding of the value of life your only alternative is to spend an unlimited amount of money in the quest for zero tolerance. You’ll still fail…typically spectacularly since any crackpot with an idea is given an opportunity to try his idea. Remember you didn’t start with a value of life so there’s no way to say the crockpots idea is crazy.  If it saves just one child…..

Can you think of an entity in this exact situation? No one willing to put a value on life and an unlimited budget (effectively)?

When zero tolerance bites you in the butt…the TSA

The federal government will never publish how much your life is worth to them, assuming they even wanted to calculate it. They can’t. It would be a political disaster.  So how can we figure out the presumed value so we see if the government expenditures are insane or not?

The TSA has a budget of roughly $7billion per year and a mandate of zero tolerance for loss of life.  Let’s assume that the worst case scenario of doing nothing is a 9/11 style attack every year (3000 dead). So what’s the value of life for an agency tasked with zero tolerance? Simple calculation really…last year the Federal government valued the life of the flying public at $7billion / 3000 or $2.4million per life. 

Just as a point of measure…42,000 people die in car crashes every year and the budget for the NHTSA is $900,000.  So the Federal government values the life of the driving public at at $900,000 / 42,000 or $21 per life. 

Think somebodies priorities are out of kilter a wee bit? Or is it airline deaths get more media attention because they are more spectacular and thus more political pressure to “do something, anything”?

When does acceptable risk come into play?

It can’t…until you put a value on life.

You willing to put a value on life? Not as easy as it sounds. But if you don’t, you’ll end up like the TSA or Medicare. In an unwinnable situation and everyone hates you.


Size is a Factor

December 8, 2010

How do you protect sensitive data and networks?  The approach you take tends to depend a lot on “size.”  For most organizations, their “size” is simply measured by sales and revenue.  For organizations processing credit cards, the “size” can be defined by the number of credit card transactions they process.  No matter what measuring stick one uses, the larger the “size” of a corporation, the more information and assets it has to protect.

The size of an entity makes decisions around what to protect become more complicated.  Most small organizations know the assets and information they need to protect, but often simply aren’t aware of  – or simply disregard – regulations specifying minimum security controls that must be in place (e.g. PCI requirements).  Large corporations tend to have the opposite problem.  They most often are well aware of the regulations that apply to them, but don’t have a clue how to strategically plan around complying with such regulations or requirements.

In a large and/or complex environment, the information and assets of an entity become exponentially more difficult to protect.  New regulations and/or requirements stipulate compliance to specific security requirements.  For large organizations and corporations that “grew up” without these stipulations, bringing themselves into compliance can be a very daunting task.  One good example is identity management.  Compliance to section 404 of Sarbanes-Oxley in the context of user provisioning, authentication, and access control can be extremely difficult for large organizations.  Legacy systems, lack of a standardized password policy, customized provisioning systems, and inefficient (yet heavily utilized) manual processes are only a few of the major challenges that an enterprise may face.

Another good example is PCI compliance.  Large organizations are charged with implementation of very specific controls to protect credit card data.  For a large enterprise that evolved without these requirements, this can be a major challenge.   Even determining where all the credit card data resides is often a herculean task, let alone architecting and implementing controls to bring the corporation into compliance.

Smaller companies face different sets of challenges.  In the case of PCI requirements, many smaller corporations are often out of PCI compliance and don’t even know it until they get a threatening letter in the mail from one of the major credit payment brands.  Once they get such a letter, it turns into a scramble to  a) figure out what PCI compliance actually is b) figure out how to comply and c) implement the controls to ensure compliance.  The smaller corporation may have an easier time determining PCI in-scope systems and environments, but often faces issues that a large corporation enterprises aren’t as concerned with.  These issues almost always involve budget and resource constraints.

Bottom line, the challenges an entity faces when attempting to protect their data and networks depends a lot on size – regardless of how it’s measured.  Larger corporations have more resources to direct at securing their data, but may have a much more difficult time implementing solutions.  Smaller organizations are much more agile, but often simply don’t have the knowledge or resources to get the job done.

 




Security Organization Redesign

November 23, 2010

Historically, security organizations have grown up organically, starting 15 – 20 years ago with a single security conscious person who ended up getting tagged with doing the job.  Over the years, that manager asked for new positions, filling a tactical need when issues were presented, creating departments/teams as it made sense. There was no particular plan in place or a long-term strategy. Eventually, you end up with more than a handful of employees and a dysfunctional team. Don’t get me wrong, the team is usually very good at putting out fires and “getting the work done” but, by no means is it robust or optimized.  They typically do not work on the issues that are most important to the company rather these large security groups are playing whack-a-mole with issues and deal with fires as they are presented.  There is no opportunity to get ahead of the game and when the fire is in your area you deal with it the best way you can.  Of course, this can cause inter-personal issues amongst the team members and duplication of efforts, driving even more dysfunction.

As a consultant, it’s easy to say that lack of planning created these problems, but I don’t know many info sec managers who could claim they have a growth plan that goes out 15 years and involves hiring 30-50 new employees.  Most security professionals, for the majority of their careers, are fighting fires

What lessons has Neohapsis learned working with our clients to reorganize their security departments?

Don’t under estimate the angst that will be voiced by the team leaders/managers within the department if they are not included in the decision making process, even if you already know the right decision.

When it comes down to it, there are only so many ways you can design a security organization.  Certain jobs and tasks make sense together.  Certain others require similar skill sets. Technically, you don’t really need to involve many people in the decision if you have someone who knows the culture of the company and has done this before. You could very easily take a CISO and a consultant and develop a new organizational structure and announce it to the CISO’s management team.  You try that, and you’ll be surprised at the uproar about not understanding the nuances of each department and the needs and issues of the individuals.  Though it will take longer, a CISO will find better acceptance with his own management team if they are allowed to go off and work together to propose an org design of their own. It will probably take 30 days and in the end, it will probably look almost identical to what the CISO and consultant would have wanted anyway.  But, the managers’ attitudes will be different and they will have buy-in.  It still doesn’t hurt to get a consultants opinion on the org design, just don’t let your management team think you outsourced their career path.  Even though you could have started your organization change 30 days ago, sometimes it is more about buy-in than being right. That’s a very hard lesson for many security professionals.

Titles are a big deal to security people

Probably the most contentious and politically painful experience, and frankly the biggest complaints from the security team leads and managers, will be coming up with proper titles for the new departments.  As is generally the case in large organizations, there are way too many Directors, VPs, Senior VPs than can honestly be justified by organizational design.  You look over the fence and wonder how everyone in the sales department can be a Senior VP.

What makes this particularly difficult within a security organization is that security professionals by nature view themselves as different or special than everyone else in the organization.  Inevitably, that means corporate HR policy is perceived to be inapplicable to them. The presumption of non-applicability is exactly what security complains about when co-workers ignore security policy. So when company policy dictates a Director title requires X number of direct reports, what do you do with your architecture group that has 5 people with 20 years security experience and no direct reports?  If you don’t title that team as Director’s or better, nobody from the outside will apply for the positions. But if you do, others in the organization will ask why there is a department of 5 people all with Director titles.

In the same vein, titles are routinely viewed by security professionals as a way of bullying co-workers into complying with a particular security policy or decision.  Any perceived lost opportunity to get a title promotion is met with severe angst, no check that…open revolt…even when no salary increase comes with it.

In the end most titles will end up being a mix of corporate policy and what levels in the organization that particular person would have to interact with (eg: need for presumed power). Yes, many feelings will be hurt.

Salaries are all over the map

In similar alignment with titles, salaries are a difficult thing to pin down in the security industry.  Sure you can go to any of a number of surveys and pull an average salary…but often they are for a generic title like “security architect” or “security analyst” or something very specific like “IDS specialist”.  Is your security analyst the same as my security analyst? I can’t tell.  Should a firewall guru get paid the same as a policy guru? Why? Why not?   Eventually you will have to look at existing salaries within the team (obviously), a third party perspective of the market conditions, and the caliber of talent you want applying.  At some level it becomes a throw a number out there and see if you a get nibble approach.

Sounds like too much work…

Are the basic issues outlined above insurmountable?  Of course not.  But they seem to be so minor that many security managers will ignore them and focus on the so called “big picture”.  Little do they know, the big picture was never really in doubt.  It was the little things that were going to give them the biggest head aches and threaten to derail the path to the big picture.

Has this happened in you organization? Did you have a re-org experience to tell? We would love comments.


Virtualization: When and where?

November 17, 2009

We often field questions from our clients regarding the risks associated with hypervisor / virtualization technology.  Ultimately the technology is still software, and still faces many of the same challenges any commercial software package faces, but there are definitely some areas worth noting.

The following thoughts are by no means a comprehensive overview of all issues, but they should provide the reader with a general foundation for thinking about virtualization-specific risks.

Generally speaking, virtual environments are not that different than physical environments.  They require much of the same care and feeding, but that’s the rub; most companies don’t do a good job of managing their physical environments, either.  Virtualization can simply make existing issues worse.

For example, if an organization doesn’t have a vulnerability management program that is effective at activities like asset identification, timely patching, maintaining the installed security technologies, change control, and system hardening, than the adoption of virtualization technology usually compounds the problem via increased “server sprawl.”  Systems become even easier to deploy which leads to more systems not being properly managed.

We often see these challenges creep up in a few scenarios:

Testing environments – Teams can get the system up and running very quickly using existing hardware.  Easy and fast…but also dirty. They often don’t take the time to harden the system or bring it up to current patch levels or install required security software.

Even in the scenarios where templates are used, with major OS vendors like Microsoft and RedHat coming out with security fixes on a monthly basis a template even 2 months old is out of date.

Rapid deployment of “utility” servers – Systems that run back-office services like mail servers, print servers, file servers, DNS servers, etc.  Often times nobody really does much custom work on them and because they can no longer be physically seen or “tripped over” in the data center they sometimes fly under the radar.

Development environments – We often see virtualization technology making inroads into companies with developers that need to spin-up and spin-down environments quickly to save time and money.  The same challenges apply; if the systems aren’t maintained (and they often aren’t – developers aren’t usually known for their attention to system administration tasks) they present great targets for the would-be attacker.  Even worse if the developers use sensitive data for testing purposes.  If properly isolated, there is less risk from what we’ve described above but that isolation has to be pretty well enforced and monitoring to really mitigate these risks.

There are also risks associated with vulnerabilities in the technology itself.  The often feared “guest break out” scenario where a virtual machine or “guest” is able to “break out” of it’s jail and take over the host (and therefore, access data in any of the other guests) is a common one, although we haven’t heard of any real-world exploitations of these defects…yet.  (Although the vulnerabilities are starting to become better understood)

There are also concerns about the hopping between security “zones” when it comes to compliance or data segregation requirement.  For example, typically a physical environment has a firewall and other security controls between a webserver and a database server.  In a virtual environment, if they are sharing the same host hardware, you typically can not put a firewall or intrusion detection device or data leakage control between them.  This could violate control mandates found in standards such as PCI in a credit card environment.

Even assuming there are no vulnerabilities in the hypervisor technology that allow for evil network games between hosts, when you house two virtual machines/guests on the same hypervisor/host you often lose the visibility of the network traffic between them.  So if your security relies on restricting or monitoring at the network level, you no longer have that ability.  Some vendors are working on solutions to resolve intra-host communication security but it’s not mature by any means.

Finally, the “many eggs in one basket” concern is still a factor; when you have 10, 20, 40 or more guest machines on a single piece of hardware that’s a lot of potential systems going down should there be a problem.  While the virtualization software vendors will certainly offer high availability scenarios with technology such as VMware’s “VMotion”, redundant hardware, the use of SANs, etc., the cost and complexity adds up fairly fast.  (And as we have seen from some rather nasty SAN failures the past two months, SANs aren’t always as failsafe as we have been lead to believe. You still have backups right?)

While in some situations the benefits of virtualization technology far outweigh the risks, there are certainly situations where existing non-virtualized architectures are better. The trick is finding that line in the midst of the hell mell towards virtualization.

–Tyler


Enterprise, or Opportunity Risk Management?

July 2, 2009

The Enterprise Risk Management (ERM) market seems to have been driven by the Finance & Banking (F&B) sector’s interpretation of what ERM means. They have taken their traditional risk methodology in the areas of Credit, Market and Operational risk management and extended that out to other areas of their businesses and called that ERM. But, for true ERM, the F&B institution’s methodology for managing risk is applying too much emphasis on backward-looking analysis of loss, as opposed to a more forward-looking speculation about potential loss (or risk) in future. Historical analysis of actual loss is of course a significant indicator of further loss in future, but only where defined losses are to be expected, such as in the areas of insurance or the provision of credit, or speculation into markets etc.

However, such a methodology doesn’t provide a sound analytical basis for those less frequent and possibly more drastic events, or those events where historical loss data doesn’t exist, which applies to the general operations of most businesses today.

So, in F&B, risk management has become very much a science, but for areas of risk outside the realms of credit, market and banking operations, and without the benefit of hindsight and a loss history, it is very much an art today.

Perhaps analyzing some of the more accepted definitions of risk will help us to figure out where and how we should be focusing our risk assessment efforts, acronyms aside:

Wikipedia by its very nature takes a broad view, not specifically in the context of business, and states simply that “risk is a concept that denoted the precise probability of specific eventualities”. Interestingly Wikipedia’s definition of risk continues by stating that risk can be defined as “the threat or probability that an action or event, will adversely or beneficially affect an organization’s ability to achieve its objectives” Hmmm, so immediately Wikipedia is recognizing that we have to tie our speculation of loss to our desire to achieve stated objectives; in other words, realizing our opportunities.

Corporate Integrity, a leading Global advisory on Governance, Risk and Compliance succinctly defines Risk as “the effect of uncertainty on business objectives”. Again, focusing on what a company is endeavouring to achieve through the realization of its opportunities.

So, how about substituting the word ENTERPRISE and replace it with OPPORTUNITY? After all, businesses are in business to profit from opportunities, and given that the above definitions of risk relate its management to the achievement of those objectives, it would seem that this has to be the basis of risk analysis.

However, this would provide us with ORM instead of ERM as an acronym for the management of risks in our business, but unfortunately, ORM is already generally accepted as meaning Operational Risk Management, which is a term well understood and accepted in the F&B world because it is a component part of the Basel II Capital Accord. This might explain the route cause of the problem!

F&B understands Operational Risk Management in Basel II terms and by extending that across the enterprise they assume it to be Enterprise Risk Management, but, as stated previously the methodology used is driven by the analysis of losses, not the analysis of risks to the achievement of objectives, goals or opportunities.

It is correct that loss analysis is an excellent way of predicting likely loss in future, but as noted earlier only if an extensive loss history exists. This is the key point. In most businesses the loss history does not exist or is very limited, and even in the F&B industry it is limited to the scope of Basel II, which does not cover business risks such as supply-chain, internal operations such as HR or reputational risks and many other risk areas.

So, whilst extensive loss history can help us to add some science to the art of risk management across the enterprise, the fact is that an extensive history of losses does not exist for most businesses, so the only viable methodology is to start with understanding a businesses strategy, its objectives, its opportunities, and trying to quantify what will prevent the company achieving those. i.e. what are the risks to the realization of opportunities.

Opportunity Risk Management (ORM).


Follow

Get every new post delivered to your Inbox.