The value of life and acceptable risk

Is it ever okay to accept the loss of life as an acceptable risk to doing business?

First off, is this even reasonable? I believe it is. Though not the best approach to calculating the cost vs benefit of a given security measure, it can be enlightening to look at past and present choices and see what they indicate about the value placed on life by how much money was spent trying to protect it.

But life is invaluable…

By value, I don’t simply mean money. Thou for simplicity sake I will use it for the rest of this post. Most people would say that life is invaluable. But that notion, though admirable and on some level true, it is not an accurate statement. Some people would trade theirs or someone elses life for cash. While others would trade their own life for another. In both cases they have, potentially unknowingly, attached a value to life…or at least a particular persons life.

It’s either valued or it’s not…

By definition, if something does not have a defined value it is either worth nothing or everything. Given that society won’t allow life to be valued at nothing (at least usually) then without a defined value, life is invaluable…or in other words…when placed in your hands, there is zero tolerance for failure to protect.

So? It’s always job number 1…

If you’ve been in the security industry for any measurable time, you will recognize the following priority list from somewhere. It shows up as the default when I do Incident Response program development.

1. Preserve life
2. Prevent physical damage to personnel, facilities, or systems
3. Prevent financial loss
…etc…

This presumes that life is the most valuable item a business/government/entity must protect. Very few, if any security professionals will face the protection of life as priority one in their career. Typically we give lip service to the priority of life since that firewall we just bought doesn’t protect life…at least directly.  But what happens when your entire reason for existing is making something safer?

Safer vs Zero tolerance

I chose the term “safer” very carefully and on purpose. It means some risk, or level of failure, is acceptable and still be considered a success. If your entire reason to exist is to protect life, you can not calculate the value of the security measure unless you know the value of life. As has been shown time and time again through history…nothing can be made perfectly safe…ala zero tolerance. But when tasked with zero tolerance, or zero breaches, and you have no understanding of the value of life your only alternative is to spend an unlimited amount of money in the quest for zero tolerance. You’ll still fail…typically spectacularly since any crackpot with an idea is given an opportunity to try his idea. Remember you didn’t start with a value of life so there’s no way to say the crockpots idea is crazy.  If it saves just one child…..

Can you think of an entity in this exact situation? No one willing to put a value on life and an unlimited budget (effectively)?

When zero tolerance bites you in the butt…the TSA

The federal government will never publish how much your life is worth to them, assuming they even wanted to calculate it. They can’t. It would be a political disaster.  So how can we figure out the presumed value so we see if the government expenditures are insane or not?

The TSA has a budget of roughly $7billion per year and a mandate of zero tolerance for loss of life.  Let’s assume that the worst case scenario of doing nothing is a 9/11 style attack every year (3000 dead). So what’s the value of life for an agency tasked with zero tolerance? Simple calculation really…last year the Federal government valued the life of the flying public at $7billion / 3000 or $2.4million per life. 

Just as a point of measure…42,000 people die in car crashes every year and the budget for the NHTSA is $900,000.  So the Federal government values the life of the driving public at at $900,000 / 42,000 or $21 per life. 

Think somebodies priorities are out of kilter a wee bit? Or is it airline deaths get more media attention because they are more spectacular and thus more political pressure to “do something, anything”?

When does acceptable risk come into play?

It can’t…until you put a value on life.

You willing to put a value on life? Not as easy as it sounds. But if you don’t, you’ll end up like the TSA or Medicare. In an unwinnable situation and everyone hates you.

Security Organization Redesign

Historically, security organizations have grown up organically, starting 15 – 20 years ago with a single security conscious person who ended up getting tagged with doing the job.  Over the years, that manager asked for new positions, filling a tactical need when issues were presented, creating departments/teams as it made sense. There was no particular plan in place or a long-term strategy. Eventually, you end up with more than a handful of employees and a dysfunctional team. Don’t get me wrong, the team is usually very good at putting out fires and “getting the work done” but, by no means is it robust or optimized.  They typically do not work on the issues that are most important to the company rather these large security groups are playing whack-a-mole with issues and deal with fires as they are presented.  There is no opportunity to get ahead of the game and when the fire is in your area you deal with it the best way you can.  Of course, this can cause inter-personal issues amongst the team members and duplication of efforts, driving even more dysfunction.

As a consultant, it’s easy to say that lack of planning created these problems, but I don’t know many info sec managers who could claim they have a growth plan that goes out 15 years and involves hiring 30-50 new employees.  Most security professionals, for the majority of their careers, are fighting fires

What lessons has Neohapsis learned working with our clients to reorganize their security departments?

Don’t under estimate the angst that will be voiced by the team leaders/managers within the department if they are not included in the decision making process, even if you already know the right decision.

When it comes down to it, there are only so many ways you can design a security organization.  Certain jobs and tasks make sense together.  Certain others require similar skill sets. Technically, you don’t really need to involve many people in the decision if you have someone who knows the culture of the company and has done this before. You could very easily take a CISO and a consultant and develop a new organizational structure and announce it to the CISO’s management team.  You try that, and you’ll be surprised at the uproar about not understanding the nuances of each department and the needs and issues of the individuals.  Though it will take longer, a CISO will find better acceptance with his own management team if they are allowed to go off and work together to propose an org design of their own. It will probably take 30 days and in the end, it will probably look almost identical to what the CISO and consultant would have wanted anyway.  But, the managers’ attitudes will be different and they will have buy-in.  It still doesn’t hurt to get a consultants opinion on the org design, just don’t let your management team think you outsourced their career path.  Even though you could have started your organization change 30 days ago, sometimes it is more about buy-in than being right. That’s a very hard lesson for many security professionals.

Titles are a big deal to security people

Probably the most contentious and politically painful experience, and frankly the biggest complaints from the security team leads and managers, will be coming up with proper titles for the new departments.  As is generally the case in large organizations, there are way too many Directors, VPs, Senior VPs than can honestly be justified by organizational design.  You look over the fence and wonder how everyone in the sales department can be a Senior VP.

What makes this particularly difficult within a security organization is that security professionals by nature view themselves as different or special than everyone else in the organization.  Inevitably, that means corporate HR policy is perceived to be inapplicable to them. The presumption of non-applicability is exactly what security complains about when co-workers ignore security policy. So when company policy dictates a Director title requires X number of direct reports, what do you do with your architecture group that has 5 people with 20 years security experience and no direct reports?  If you don’t title that team as Director’s or better, nobody from the outside will apply for the positions. But if you do, others in the organization will ask why there is a department of 5 people all with Director titles.

In the same vein, titles are routinely viewed by security professionals as a way of bullying co-workers into complying with a particular security policy or decision.  Any perceived lost opportunity to get a title promotion is met with severe angst, no check that…open revolt…even when no salary increase comes with it.

In the end most titles will end up being a mix of corporate policy and what levels in the organization that particular person would have to interact with (eg: need for presumed power). Yes, many feelings will be hurt.

Salaries are all over the map

In similar alignment with titles, salaries are a difficult thing to pin down in the security industry.  Sure you can go to any of a number of surveys and pull an average salary…but often they are for a generic title like “security architect” or “security analyst” or something very specific like “IDS specialist”.  Is your security analyst the same as my security analyst? I can’t tell.  Should a firewall guru get paid the same as a policy guru? Why? Why not?   Eventually you will have to look at existing salaries within the team (obviously), a third party perspective of the market conditions, and the caliber of talent you want applying.  At some level it becomes a throw a number out there and see if you a get nibble approach.

Sounds like too much work…

Are the basic issues outlined above insurmountable?  Of course not.  But they seem to be so minor that many security managers will ignore them and focus on the so called “big picture”.  Little do they know, the big picture was never really in doubt.  It was the little things that were going to give them the biggest head aches and threaten to derail the path to the big picture.

Has this happened in you organization? Did you have a re-org experience to tell? We would love comments.

Virtualization: When and where?

We often field questions from our clients regarding the risks associated with hypervisor / virtualization technology.  Ultimately the technology is still software, and still faces many of the same challenges any commercial software package faces, but there are definitely some areas worth noting.

The following thoughts are by no means a comprehensive overview of all issues, but they should provide the reader with a general foundation for thinking about virtualization-specific risks.

Generally speaking, virtual environments are not that different than physical environments.  They require much of the same care and feeding, but that’s the rub; most companies don’t do a good job of managing their physical environments, either.  Virtualization can simply make existing issues worse.

For example, if an organization doesn’t have a vulnerability management program that is effective at activities like asset identification, timely patching, maintaining the installed security technologies, change control, and system hardening, than the adoption of virtualization technology usually compounds the problem via increased “server sprawl.”  Systems become even easier to deploy which leads to more systems not being properly managed.

We often see these challenges creep up in a few scenarios:

Testing environments – Teams can get the system up and running very quickly using existing hardware.  Easy and fast…but also dirty. They often don’t take the time to harden the system or bring it up to current patch levels or install required security software.

Even in the scenarios where templates are used, with major OS vendors like Microsoft and RedHat coming out with security fixes on a monthly basis a template even 2 months old is out of date.

Rapid deployment of “utility” servers – Systems that run back-office services like mail servers, print servers, file servers, DNS servers, etc.  Often times nobody really does much custom work on them and because they can no longer be physically seen or “tripped over” in the data center they sometimes fly under the radar.

Development environments – We often see virtualization technology making inroads into companies with developers that need to spin-up and spin-down environments quickly to save time and money.  The same challenges apply; if the systems aren’t maintained (and they often aren’t – developers aren’t usually known for their attention to system administration tasks) they present great targets for the would-be attacker.  Even worse if the developers use sensitive data for testing purposes.  If properly isolated, there is less risk from what we’ve described above but that isolation has to be pretty well enforced and monitoring to really mitigate these risks.

There are also risks associated with vulnerabilities in the technology itself.  The often feared “guest break out” scenario where a virtual machine or “guest” is able to “break out” of it’s jail and take over the host (and therefore, access data in any of the other guests) is a common one, although we haven’t heard of any real-world exploitations of these defects…yet.  (Although the vulnerabilities are starting to become better understood)

There are also concerns about the hopping between security “zones” when it comes to compliance or data segregation requirement.  For example, typically a physical environment has a firewall and other security controls between a webserver and a database server.  In a virtual environment, if they are sharing the same host hardware, you typically can not put a firewall or intrusion detection device or data leakage control between them.  This could violate control mandates found in standards such as PCI in a credit card environment.

Even assuming there are no vulnerabilities in the hypervisor technology that allow for evil network games between hosts, when you house two virtual machines/guests on the same hypervisor/host you often lose the visibility of the network traffic between them.  So if your security relies on restricting or monitoring at the network level, you no longer have that ability.  Some vendors are working on solutions to resolve intra-host communication security but it’s not mature by any means.

Finally, the “many eggs in one basket” concern is still a factor; when you have 10, 20, 40 or more guest machines on a single piece of hardware that’s a lot of potential systems going down should there be a problem.  While the virtualization software vendors will certainly offer high availability scenarios with technology such as VMware’s “VMotion”, redundant hardware, the use of SANs, etc., the cost and complexity adds up fairly fast.  (And as we have seen from some rather nasty SAN failures the past two months, SANs aren’t always as failsafe as we have been lead to believe. You still have backups right?)

While in some situations the benefits of virtualization technology far outweigh the risks, there are certainly situations where existing non-virtualized architectures are better. The trick is finding that line in the midst of the hell mell towards virtualization.

–Tyler