All Your Virtualized PCI Compliance Are Belong To Us...

Another interesting example I use in my VirtSec presentations when discussing the challenges of what I describe as Phase 2 of virtualization -- virtualizing critical applications and things like Internet-facing infrastructure in DMZ's -- is the notion of compliance failures based on existing and upcoming revisions to regulatory requirements.

Specifically, I use PCI/DSS to illustrate that in many cases were one to take a highly-segmented and stratified "defense-in-depth" architecture that is today "PCI compliant" and virtualize it given presently available options, you'd likely find yourself out of compliance given the current state of technology solutions and auditing standards used to assess against.

Then again, you might just pass with flying colors while being totally insecure.

Here's a fantastic example from Eric Siebert over at the TechTarget Virtualization blog.  Check this out, it's a doozie!

Having just survived another annual PCI compliance audit, I was again surprised that the strict standards for securing servers that must be followed contain nothing specific concerning virtual hosts and networks. Our auditor focused on guest virtual machines (VMs), ensuring they had up-to-date patches, locked-down security settings and current anti-virus definitions. But ironically, the host server that the virtual machines were running on went completely ignored. If the host server was compromised, it wouldn’t matter how secure the VMs were because they could be easily accessed. Host servers should always be securely locked down to protect the VMs which are running on them.

It seems that much of the IT industry has yet to react to the virtualization trend, having been slow in changing procedures to adjust to some of the unconventional concepts that virtualization introduces. When I told our auditor that the servers were virtual, the only thing he wanted to see was some documentation stating that the remote console sessions to the VMs were secure. It’s probably just a matter of time before specific requirements for virtual servers are introduced. In fact, a recent webinar takes up this issue of whether or not virtualized servers can be considered compliant, addressing section 2.2.1 of the PCI DSS which states, “Implement only one primary function per server”; that is to say, web servers, database servers and DNS should be implemented on separate servers. Virtual servers typically have many functions running on a single physical server, which would make them noncompliant.

So let's assume that what Eric talks about in section 2.2.1 of PCI/DSS holds true, that basically means two things: (1) PCI/DSS intimates that virtualization cannot provide the same level of security as non-virtualized infrastructure and (2) you won't be able to virtualize infrastructure governed by PCI/DSS if you expect to be compliant.

Now, this goes toward the stuff Mogull and I were talking about in terms of assessing risk and using the notion of "zone defense" for asset segmentation in virtualized infrastructure. 

Here's a snippet from my VirtSec preso on the point:

Further, as I mentioned in my post titled "Risky Business -- The Next Audit Cycle: Bellweather Test for Critical Production Virtualized Infrastructure," this next audit cycle is going to be interesting for many companies...

Yippeee!

/Hoff

Comments

[Michael Janke]

Extend those thoughts just a bit, on the thread of 'loss of sensitive data', and include not only the hosting server, but also the clones and copies of virtualized servers that tend to be laying around the infrastructure. In some cases, such as DR or business continuity, we want copies of VM's stored off line or of site to achieve desired RPO/RTO goals. In other cases, the copies are left laying around for the convenience of the server manager. Those copies are a potential security problem, just like loose backup tapes. Releasing a copy of sensitive data from a cloned VM that is stored off to the side 'just in case' generates the same headlines as releasing the original, but because the copies are scattered about uncatalogued on various storage devices, the probability of accidental release is higher.

Ding! - I now have to secure every clone of every VM just the same as the original host, and to do that I need security and auditing and change control on the spot where I store the 'spare' VM's that is at least equal to the security requirements of the most sensitive host copy that I store.

So if I want to hack a datacenter and grab interesting data from a server, I have the virtualized server containing the data as a target, the host server that is hosting the live virtual server as a target, the tape backups (or backup-to-disk-pool) as a target, and the big cheap NAS device where the sys admin stores copies of the VM's off to the side 'just in case'. I'd aim for the latter first.

Hmmm.. will the PCI auditor ask me where all my clones are stored?

[Christofer Hoff]

@michael:

Better yet (and in one of my slides) is the fact that by design and roadmap progression these files are almost all stored on centralized SAN clusters.

I've asked many times how many of the security pros in my audiences are storage security experts -- nobody has yet raised their hand.

You get access to the SAN -- as you point out -- not only do you get the OS and the applications, you get the data that goes with it...all in one nice tidy bundle!

I'll be posting that slide shortly.

/Hoff

[Michael Janke]

OK - I was thinking not of access to the SAN itself, but rather access to the do-it-all 'utility' servers that system admins have in the back corner and use to store their tool kits, scripts and ummm... spare VM clones. You know, the servers that nobody knows the real name of, that don't show up as a part of any application & that the security and audit dudes don't even know exist. Those servers. They happen to be SAN attached, and happen to have a gold mine of interesting bits on them.

But now that you bring up the SAN itself, the lack of overlap between security people, who tend to be mostly network people, and the storage teams, who probably have never seen a hacked server, is a big issue. Unfortunately there is no overlap.

When you post you SAN slide, I'll have more comments. ;)

[colin]

Christofer,

This raises some points that I have been trying to raise for some time with various organisations that are virtualising at a great rate of knots - how does your virtualisation strategy affect your regulatory and compliance goals?

This is invariably met with show gazing, various mumbling sounds and eventual shuffling away. The amazing thing is that this exact scenario is played out when asking other QSA's about how to assess security posture in a virtualised environment.

The PCI consortium have not specifically addressed virtualisation as it is a concept that the card processors themselves have not embraced in any way. I believe the approach to a stratified architecture is almost completely lost when mixing assets of multiple classifications under a lone hypervisor.

I think some of the compliance goals can be met if systems of the same classification can live on the same hypervisor, but as one of your commentors has pointed out, the back end data storage again has the same issues. Multiple classifications of data residing on the one storage medium, to my mind, would be an immediate non compliance.

I was recently asked about compliance and PCI/ACSI 33 with regards virtualisation. I hate to admit it, but my response was "if you are attempting to meet something as stringent as ACSI 33, just don't virtualise".

-colin.

[LonerVamp]

Could one intepret the PCI rule to say the VM host server serves the one function of hosting VMs? I guess this might contrast with the answer to the question of whether the PCI-related web server can host multiple web sites or a website that does multiple functions? This is an innocent question; I don't know the answer to either of those.

I posted other ancillary thoughts on my own blog. :)

[B.K. DeLong]

Well, the basic rule of PCI is that any part of the network that credit card info goes over or is stored on is in-scope. Network segmentation on to separate, secure VLANs is the best way to reduce scope so therefore it stands to reason that if any VM on a host server contains CC info then said host servers (and all VMs on it) become in-scope.

I think I'm too close to the issue as treating each host server as its own "VLAN" seems like a no-brainer to me. It's clearly not a best-practice to host VMs that run across multiple secure zones yet...it seems to be fairly prevalent.

Is it simply a matter of convenience vs security? i.e. companies try to consolidate as much as possible even if it means crossing zones? Sounds pretty risky with all the unknowns regarding virtualization security.

[LonerVamp]

It's a matter of convenience/cost vs security for us. We are tinkering with our need for a DMZ so that we have less zones to keep VM infrastructure in. That or keep a DMZ and fully separate our zones. The latter is expensive, though.

I think we'll end up going expensive but separate, but we do have to check out our options since it also means more work for us as moving VMs between zones will become less easy.