Service Provider Scoping Angst

July 15, 2011

By Patrick Harbauer

Over the past several months we have had many service providers come to us wringing their hands wondering if they should go through the ROC process. They may offer cloud services or a managed security service, for example, and they keep running up against QSA’s peppering them with questions when the QSA is assessing one of the service provider’s customers. Or the sales cycle repeatedly hits road blocks when a potential customer’s InfoSec department raises a red flag and wants to know what security controls the service provider has in place to protect their customers from a security breach.

I equate this dilemma that service providers face to going to the hardware store to buy an extension cord. If the hardware store carries extension cords from four manufactures, three of the manufactures have “UL Listed” on their packaging and one has “Not UL Listed” on their packaging, would anyone buy the “Not UL Listed” extension cord? Absolutely not!

So, if the service provider decides to engage us to pursue PCI compliance, they begin to wring their hands again when trying to determine what is in scope. Client employees, and sometimes even the individuals who hired us, ask “Why are we going through this again? We don’t process, store or transmit payment card information?!?!?!??”

Again, I turn to the UL analogy. The extension cord manufacturer may wonder if they should certify the male end of the extension cord, the female end or just the wire.” They could spin their wheels for a long time trying to determine what the scope should be. This is where service providers need to leverage their QSA’s expertise. Again, using my extension cord analogy (OK, so maybe it’s a little goofy but it works for me!!) UL will most likely say, “Mr. Customer, if we are going to put our logo on this product, we have to use our best judgment and examine all aspects of this extension cord to make sure that it is safe to use and won’t cause a fire that results in one or more deaths. If you are not willing to let us perform what, in our best judgment, is a thorough set of testing procedures on any aspect of your product that we deem necessary, we cannot certify your product.”

As QSA’s assessing a service provider’s products and services for PCI compliance, we must be allowed to use our best judgment to determine what components of a service provider’s environment are in scope and the PCI requirements and testing procedures that are applicable. It is our responsibility as QSA’s to make sure that a merchant doesn’t “burn their business” as the result of making the service provider’s solution a part of their CDE only to find out that the solution is not properly secured in the spirit of the PCI DSS.


PCI Surprises

January 31, 2011

By Patrick Harbauer

Whenever we perform a PCI assessment for a new client, we invariably have the Gomer Pyle “Surprise!, surprise!” conversation with IT management. And the outcome of the conversation is that IT security controls are more closely monitored and the overall security posture of the organization improves. Here are a few examples:

Swiss Cheese Firewalls – When we perform a PCI assessment, we take firewall rule set reviews very seriously. Besides finding the obvious “ANY ANY” rule infractions, we find rules that were meant to be “temporary for testing” or rules that are no longer needed because entire environments have been decommissioned. It isn’t uncommon to see 10-20% of the firewall rules removed or tightened to allow only protocols, IP’s and ports that are actually required for the cardholder environment to function properly.

Missing Patches – Time and again we find in-scope systems that are not properly patched. This is usually due to overtaxed IT staff who don’t find the time to patch systems or a malfunctioning patching software solution. And in some cases administrators have been burned by a patch that brought down an application and vow to never patch again. What we usually find is that “practice makes perfect” with patching. Organizations that are up to date on patches have well-defined processes and document procedures to perform patching. And that leads us to our next issue…

One Environment – In many cases, organizations that are not up-to-date with their patches do not have isolated test/development and production environments. Besides being a PCI violation to have test/development and production systems on the same network and/or servers, if you do not have a test environment that mirrors production, you are more likely to break production applications when you patch. You will be much more successful remaining current with patches if you have a test environment that mirrors production and where you can address issues before applying the patches to production systems.

These are just a few examples of what we see when performing PCI assessments for new clients and illustrates some of the benefits that come out of a PCI assessment.


Virtualization: When and where?

November 17, 2009

We often field questions from our clients regarding the risks associated with hypervisor / virtualization technology.  Ultimately the technology is still software, and still faces many of the same challenges any commercial software package faces, but there are definitely some areas worth noting.

The following thoughts are by no means a comprehensive overview of all issues, but they should provide the reader with a general foundation for thinking about virtualization-specific risks.

Generally speaking, virtual environments are not that different than physical environments.  They require much of the same care and feeding, but that’s the rub; most companies don’t do a good job of managing their physical environments, either.  Virtualization can simply make existing issues worse.

For example, if an organization doesn’t have a vulnerability management program that is effective at activities like asset identification, timely patching, maintaining the installed security technologies, change control, and system hardening, than the adoption of virtualization technology usually compounds the problem via increased “server sprawl.”  Systems become even easier to deploy which leads to more systems not being properly managed.

We often see these challenges creep up in a few scenarios:

Testing environments – Teams can get the system up and running very quickly using existing hardware.  Easy and fast…but also dirty. They often don’t take the time to harden the system or bring it up to current patch levels or install required security software.

Even in the scenarios where templates are used, with major OS vendors like Microsoft and RedHat coming out with security fixes on a monthly basis a template even 2 months old is out of date.

Rapid deployment of “utility” servers – Systems that run back-office services like mail servers, print servers, file servers, DNS servers, etc.  Often times nobody really does much custom work on them and because they can no longer be physically seen or “tripped over” in the data center they sometimes fly under the radar.

Development environments – We often see virtualization technology making inroads into companies with developers that need to spin-up and spin-down environments quickly to save time and money.  The same challenges apply; if the systems aren’t maintained (and they often aren’t – developers aren’t usually known for their attention to system administration tasks) they present great targets for the would-be attacker.  Even worse if the developers use sensitive data for testing purposes.  If properly isolated, there is less risk from what we’ve described above but that isolation has to be pretty well enforced and monitoring to really mitigate these risks.

There are also risks associated with vulnerabilities in the technology itself.  The often feared “guest break out” scenario where a virtual machine or “guest” is able to “break out” of it’s jail and take over the host (and therefore, access data in any of the other guests) is a common one, although we haven’t heard of any real-world exploitations of these defects…yet.  (Although the vulnerabilities are starting to become better understood)

There are also concerns about the hopping between security “zones” when it comes to compliance or data segregation requirement.  For example, typically a physical environment has a firewall and other security controls between a webserver and a database server.  In a virtual environment, if they are sharing the same host hardware, you typically can not put a firewall or intrusion detection device or data leakage control between them.  This could violate control mandates found in standards such as PCI in a credit card environment.

Even assuming there are no vulnerabilities in the hypervisor technology that allow for evil network games between hosts, when you house two virtual machines/guests on the same hypervisor/host you often lose the visibility of the network traffic between them.  So if your security relies on restricting or monitoring at the network level, you no longer have that ability.  Some vendors are working on solutions to resolve intra-host communication security but it’s not mature by any means.

Finally, the “many eggs in one basket” concern is still a factor; when you have 10, 20, 40 or more guest machines on a single piece of hardware that’s a lot of potential systems going down should there be a problem.  While the virtualization software vendors will certainly offer high availability scenarios with technology such as VMware’s “VMotion”, redundant hardware, the use of SANs, etc., the cost and complexity adds up fairly fast.  (And as we have seen from some rather nasty SAN failures the past two months, SANs aren’t always as failsafe as we have been lead to believe. You still have backups right?)

While in some situations the benefits of virtualization technology far outweigh the risks, there are certainly situations where existing non-virtualized architectures are better. The trick is finding that line in the midst of the hell mell towards virtualization.

–Tyler


Response to Visa’s Chief Enterprise Risk Officer comments on PCI DSS

March 27, 2009

Visa’s Chief Enterprise Risk Officer, Ellen Richey, recently presented at the Visa Security Summit on March 19th. One of the valuable points made in her presentation was defending the value of implementing PCI DSS to protect against data theft. In addition, Ellen Richey spoke about the challenge organizations face, not only becoming compliant, but proactively maintaining compliance, defending against attacks and protecting sensitive information.

Recent compromises of payment processors and merchants that were stated to be PCI compliant have brought criticism to the PCI program. Our views are strongly aligned with the views presented by Ellen Richey. While the current PCI program requires an annual audit, this audit is simply an annual health-check. If you were to view the PCI audit like a state vehicle inspection. Even though at the time of the inspection everything on your car checks out, this does not prevent the situation of days later your brake lights go out. You would still have a valid inspection sticker, but are no longer in compliance with safety requirements. It is the owner’s responsibility to ensure the car is maintained appropriately. Similarly in PCI, it is the company’s responsibility to ensure the effectiveness and maintenance of controls to protect their data in an ongoing manner.

Ellen Richey also mentioned increased collaboration with the payment card industry, merchants and consumers. Collaboration is a key step to implementing the technology and processes necessary to continue reducing fraud and data theft. From a merchant, service provider and payment processor perspective, new technologies and programs will continue to reduce transaction risk, but, today, there are areas where these organizations need to proactively improve. The PCI DSS standard provides guidance around the implementation of controls to protect data. Though in addition to protecting data, merchants, service providers and processors need to proactively address their ability to detect attack and be prepared to respond effectively in the event of a compromise. These are two areas that are not currently adequately addressed by the PCI DSS and are areas where we continue to see organizations lacking.

See the following link to the Remarks by Ellen Richey, Chief Enterprise Risk Officer, Visa Inc. at the Visa Security Summit, March 19, 2009:

http://www.corporate.visa.com/md/dl/documents/downloads/EllenRichey09SummitRemarks.pdf


Follow

Get every new post delivered to your Inbox.