Rational Survivability

The Final Frontier(?): Virtualizing the DMZ...

Alessandro from virtualization.info and I were chatting today regarding VMware's latest best-practices document titled "DMZ Virtualization with VMware Infrastructure." 

This is a nine page overview that does a reasonably good job of laying out many of the architectural/topological options available when thinking about taking the steps toward virtualizing what some consider the "final frontier" in the proving grounds of production-level virtualization -- the (Internet-facing) DMZ.

The whitepaper was timely because I was just finishing up my presentation for Blackhat and was busy creating a similar set of high-level architectural examples to use in my presentation.  I decided to reference those in the document because they quite elegantly represent the starting points that many folks would use as a stepping off point in their virtual DMZ adventures.

...and I think it will be an adventure punctuated perhaps by evolutionary steps as documented in the options presented in the whitepaper.

As I read through the document, I had to remind myself of the fact that this was intended to be a high-level document and not designed to cover the hairy edges of network and security design. 

The whitepaper highlighted some of the reasonable trade-off's in complexity, resiliency, management, functionality, operational expertise, and cost but given where my head and focus are today, I have to admit that it still gnawed at me from a security perspective which is still too weak for my liking.

I've hinted at why in my original Four Horsemen slide, and I'm going to be speaking for 75 minutes on the topic at Blackhat, so come get your VirtSec boogie on there for a full explanation...

Alessandro got dinged in a comment on his blog for a statement in which he suggested that partially-collapsed as well as fully-collapsed DMZ's with virtual separation of trust zones "...should be avoided at all costs because they imply the inviolability of the hypervisor (at any level: from the virtual networking to the kernel) something that nor VMware neither any other virtualization vendor can grant."

This appears contradictory to his initial assessment of DMZ virtualization wherein he stated that "...there [is] nothing bad in virtualizing the DMZ as long as we are fully aware of the risks."  In a way, I think I understand exactly where Alessandro is coming from, even if I don't completely agree with him (or at least I partially do...)

This really paints an altogether unfortunate and yet accurate picture of the circular arguments folks engage in when they combine the following topics in a single argument:

• Securing virtualization
• Virtualizing security
• Security via virtualization

In the same way that we trust our operating system vendors who provide us with the operational underpinnings of our datacenters with the hope that they will approach a reasonable level of "security" in their products, we are basically at the same point with our virtualization (OS) platform providers.

Hope is not a strategy, but it seems we've at least accepted it for the time being... ;(

Sure there are new attack vectors and operational risks, but the slippery slope of not being able to really quantify whether you are more or less at risk based solely on the one-dimensional data point of the infallibility of the hypervisor  and then write the whole concept off seems a little odd to me.

If you're truly assessing risk in the potential virtualization of your DMZ, you'll take the operational/architectural guidelines as well as the subjective business impacts into consideration.  Simply stating that one should or should not virtualize a DMZ without a holistic approach is myopic.

To circle back on the topic, the choice of whether to -- and how to -- virtualize your DMZ  is really starting to gain traction.  I think the whitepaper took a decent first-pass stab at exploring how one might approach it, but the devil's in the details -- or at least the devil's 4 horsemen are ;)

/Hoff

Blackhat 2008: Four Horsemen Of the Virtualization Apocalypse - Done!

Today was the deadline for submission for all selected Blackhat presentations. 

I'm giving a 75 minute talk titled "The Four Horsemen of the Virtualization Apocalypse" which is based upon my original blog posting here.

I dutifully uploaded my presentation to Ping and the gang at Blackhat HQ today (on time, that's a first!) with a sigh of relief and accomplishment.  I've done hundreds of presentations over the years, but this one is special.

I have to say that I poured my heart and soul into this presentation.  I went all "Zen and the Art of Presentation" for most of it and I think that combined with the dozens of hours I put into the content, the diagrams and animations turned out purdy. ;)

Once BH is done, I'll be posting it online with my narrative as I have my other presentations.

This cathartic little post is just the final little exhale of this project prior to numerous advance rehearsals, the first of which I will be inflicting upon my unwitting guests (75+ of them thus far) at my July 5th Pig Roast & Mojito festival in honor of another notch in the annual belt I've managed to stay alive on this hunk o' rock.

Speaking of which, if you're in the MA area and want an amazing cuban or southern-style pulled pork feast with all the trimmings, drop me a line as everyone's welcome...many of the BeanSec'rs are coming, you should too!

Happy 4th/5th!

/Hoff

VirtSec Not A Market!? Fugghetaboutit!

Thanks to Alan Shimel and his pre-Blackhat Security Bloggers Network commentary, a bunch of interesting folks are commenting on the topic of virtualization security (VirtSec) which is the focus of my preso at Blackhat this year.

Mike Rothman did his part this morning by writing up a thought-provoking piece opining on the lack of a near-term market for VirtSec solutions:

So I'm not going to talk about technical stuff. Yet, I do feel compelled to draw the conclusion that despite the dangers, it doesn't matter. All the folks that are trying to make VirtSec into a market are basically just pushing on a rope.

That's right. Now matter how hard you push (or how many blog postings you write), you are not going to make VirtSec into a market for at least 2 years. And that is being pretty optimistic. So for all those VCs that are thinking they've jumped onto the next big security opportunity, I hope your partnership will allow you to be patient.

Again, it's not because the risks of virtualization aren't real. If guys like Hoff and Thomas say they are, then I tend to believe them. But Mr. Market doesn't care what smart guys say. Mr. Market cares about budget cycles and priorities and political affiliations, and none of these lead me to believe that VirtSec revenues are going to accelerate anytime soon.

Firstly, almost all markets take a couple of years to fully develop and mature and VirtSec is no different.  Nobody said that VirtSec will violate the laws of physics, but it's also a very hot topic and consumers/adopters are recognizing that security is a piece of the puzzle that is missing.

In many cases this is because virtualization platform providers have simply marketed virtualization as being "as secure" or "more secure" than than their physical counterparts.  This, combined with the rapid adoption of virtualization, has caused a knee jerk reactive reaction.

By the way, this is completely par for the course in our industry.  If you act surprised, you deserve an Emmy ;)

Secondly, and most importantly to me, Mike did me a bit of a disservice by intimating that my pushing the issues regarding VirtSec are focused solely on the technical.  Sadly, that's so far off base from my "fair and balanced" perspective on the matter because along with the technical issues, I constantly drum home the following:

• The biggest challenge we have with virtualization in the short term (the next couple of years -- the same timeframe Mike suggests VirtSec will take to percolate) is operational and organizational, and not technical.
  
  You can find recent postings on that topic in posts such as Security Pros Say VirtSec Is An Operations Problem? and The Challenge of Virtualization Security: Organizational and Operational, NOT Technical   
  
  
  
• Along with the organizational and operational issues comes the need for visibility and visualization of what is going on within these virtual environments.
  
  You can find my latest entry on this (posted early today, actually) in my entry titled Visualization Through Virtualization...

"Nobody Puts Baby In the Corner"

Painting only one of the legs of the stool as my sole argument isn't accurate and doesn't portray what I have been talking about for some time -- and agree with Mike about -- that these challenges are more than one-dimensional.

The reality is that Mike is right -- the budget, priority and politics will bracket VirtSec's adoption, but only if you think of VirtSec as a technical problem.

Is VirtSec a market?  My opinion: it's an instantiation of technology, practice and operational adjustment brought forth as a derivative of a disruptive technology and prevailing market conditions. 

Does that mean it's a feature as opposed to a market?  No.  In my opinion, it's an evolution of an existing market, rife with existing solutions and punctuated by emerging ones.

The next stop is how "security" will evolve from VirtSec to CloudSec...

/Hoff

Visualization Through Virtualization...

I've spent quite a bit of time investigating emerging technology solutions for virtualization security (VirtSec) lately.  I've made mention of an idea that conceptually didn't gel until this last week.

I was speaking at TechTarget's Financial Information Security Decisions show in New York and was paired up in the network workshop with Joel Snyder of Opus One.

Joel was presenting his 5 myths of Information Security and one of the myths was (paraphrasing) that Intrusion Detection solutions don't detect solutions. 

What Joel went on to suggest is that what IDS solutions actually do is provide one with a perspective visibility across the network; determining what represents an actual "intrusion" is a contextual argument that goes to the efficacy and correlation capabilities of the platform(s.)

This got me thinking along the lines of some of the emerging IDP (intrusion detection and prevention) solutions from emerging vendors in the virtualization space.

Something rather profound but obvious dawned on me.

Given the integration for management of these "security" solutions with the management platforms of the virtualization platform providers AND the operational shift of who was managing the security solutions (see here) really means that these aren't really virtualization security solutions at all, they are actually vitualization visualization solutions.

Virtualization management platforms provide the configuration and operational telemetry regarding the virtual environment to these solutions which does what most HostSec or NetSec solutions have been unable to do in the past: gain context regarding how the infrastructure the security solutions are protecting are actually configured.

HostSec and NetSec solutions have no context of the solutions outside of the host they are protecting or the network segment/IP address they are connected to respectively.  Not so with VirtSec solutions.

That's pretty neat when you think of it.  Even though we're substantially handicapped as to what these solutions can *do* with this capability today (see here) integrating this capability can dramatically and positively affect the way in which "security" administration and analytics manifests themselves over time.

"Yeah, but these are basically the same views someone might get looking at a firewall, IDS or IPS tool today," you might argue.  That's right, except we already know that server and virtualization administrators (as well as most network folk) don't have access to those tools...

So in many cases the administrators who will be looking at this information are not "security" folks by trade, so the (and you'll excuse the wording) dumbing down of this information actually provides a very good perch upon which to troubleshoot and extend the forced simplicity of "checkbox" security in the virtualization platforms to this new class of security administrator.

This may be the first time some of these teams have had access to "security" telemetry of this kind.

In the long term, he challenge will be how, when you have multiple of these solutions, you gain a consolidated view, but the reality is that the NetSec and HostSec admins can use this same view and then click-through into the specific toolset management stacks for finer-grained configuration/analysis. 

This is actually an interesting way to think about how the re-integration of the server admins, network and security teams might become more cohesive operationally in the future...through the same lens of visualizing the environment.

Here are some ideas of what I'm talking about; these are some snapshots of management interfaces of upcoming VirtSec solution providers.  These are random shots of some of the different views of managing virtual appliances...

Altor:

Blue Lane:

Catbird:

Reflex:

Thanks to Amir-Ben Afraim (Altor,) Greg Ness (Blue Lane,) Michael Berman (Catbird,) and Dave Devalk (Reflex) for getting these images to me.  Also, hat-tip to Joel Snyder for the noodle nudge...

/Hoff

Self Healing Intrusion Tolerance...

Tim Greene from Computerworld wrote a story last week titled "Security software makes virtual servers a moving target." 

This story draws attention to a story on the same topic that popped up a while ago (see Dark Reading) about some research led by George Mason University professor Arun Sood that is being productized and marketed as "Self Cleansing Intrusion Tolerance (SCIT)"

SCIT is based upon the premise that taking machines (within a virtualized environment) in and out of service rapidly and additionally substituting the underlying operating systems/application combinations reduces the exposure of attack and hastens the remediation/mitigation process by introducing the notion of what Sood calls "security by diversity."

Examples are given in the article suggesting the applicability of application types for SCIT:

SCIT is best suited to servers with short transaction times and has been tested with DNS, Web and single-sign-on servers, he says, which can perform effectively even if each virtual server is in use for just seconds.

In today's multi-tier, SOA, web2.0, cloud-compute, mashup world, with or without the issue of preservation of state across even short-transactional applications, I'm not sure I see the practical utility in this approach.  The high-level concept, yes, the underlying operational reality...not so much.

Some of you might notice the, um, slightly different comparative version of Sood's acronym reflecting my opinion of this approach in this blog entry's title... ;)

I think that SCIT's underlying principles lend themselves well to the notions I champion of resilient and survivable systems, but I think that the mechanical practicality of the proposed solutions -- even within the highly dynamic and agile framework of virtualization -- simply aren't realistic today.

Real-time infrastructure with it's dynamic orchestration, provisioning, governance, and security is certainly evolving and we might get to the point where heterogeneous systems are autonomously secured based upon global policy definitions up and down the stack, but we are quite some time away from being able to realize this vision.

You will no doubt notice that the focal element of SCIT is the concept of a security-centric perspective on lifecycle management of VM's.  It's quite obvious that VM lifecycle management is a hotly-contested topic for which many of the large infrastructure players are battling. 

Security will simply be a piece of this puzzle, not the focus of it.

This is not to say that this solution is not worthy of consideration as we look out across the horizon, and from a timing perspective it will likely surface again given it's "ahead of it's deployable time" status but I'm forced to consider what box I'd check in describing SCIT today:

• Feature
• Solution
• Future

Neat stuff, but if you're going to take investment and productize something, it's got to be realistically deployable.  I'd suggest that baking this sort of functionality into the virtualization platforms themselves and allowing for universal telemetry (sort of like this) to allow for either "self cleansing intrusion tolerance" or even "self healing intrusion tolerance" is probably a more reasonable concept. 

/Hoff

Security Pros Say VirtSec Is An Operations Problem?

Mark Gaydos from Tripwire's blog wrote an interesting article titled "Ops or Security: Who’s Responsible for Securing Virtualization?"  The outcome is pretty much inline with my prior points that the biggest challenges we have in virtualization are operational and organizational rather than technical.

To wit, I quoteth from Mark's post:

Tripwire recently performed a 25 question survey on virtualization security.  Respondents broke down 78%/22% between management and administrator/staff respectively.  We will be publishing a report around this survey in the next two weeks. 

However, one of the interesting points that came out of the survey was that respondents feel that the operations team is responsible for securing a virtualized environment (almost two thirds of the respondents felt this way).  This includes over half of the actual  “security” personnel who took the survey who feel operations has this responsibility. 

That’s right!  Over half of the people covering security who responded to the survey said operations needs to secure virtual systems and not them.

My question is why?  Does security not want to deal with virtualization?  Do personnel feel that operations is closer to virtualization and they understand the issues?  Does security just want to wash their hands of the issue?  Or is management just leaning towards having operations handle everything around virtualization?

However, I wonder how much Mark read into the security personnel's answers inasmuch as he suggests that they do "...not want to deal with virtualization" versus perhaps the fact that they don't actually have the visibility or access to the tools to do so!*

Responsibility versus desire are two very different things!

Managing the "security" of virtualized environments today really centers around the deployments of virtual appliances and the configuration of the vSwitches.  That means in a VMware environment, you have to have access and rights via Virtualcenter.  The same is true in terms of Xen derivatives; if you don't have access to configure and provision the networking and VM's, you're done.

Security in virtualized environments today is literally often thought of as a checkbox or two in a GUI somewhere.  (All things considered, it would be great to be able to realize that one day...)

Just like security folks have locked server and network admins out of *their* firewalls and IPS's, and as network folks have done the same in *their* routers and switches, virtual SysAdmins have done the same in *their* virtual server environments.  If you don't have access to the VM command and control, you can't manage the security bits and pieces bolted onto it.

I don't think it's that the security folks *want* to surrender the responsibility, I think it's that they never had it in the first place the moment the V-word entered the picture.

It ain't rocket science.  It ain't voodoo.  It ain't a tectonic buck-passing conspiracy.  It's access, separation of duties (by force,) visibility and capability, plain and simple.

/Hoff

*Update: Per Amrit's excellent comments, I look forward to Tripwire releasing the report to gain clarity on the question(s) asked as it begs the point as to whether the answers Mark refers to were in regards to the mechanical operationalization of security (the "doing" part) or the policy, strategy, audit and monitoring  tasks.  Are we talking about "security management" in general or "security operations?"

In either circumstance the "security" team is -- based upon my observation from feedback -- being left out of both.

Get Tripwire's ConfigCheck For VMware ESX...

From my good friends over at Tripwire...

I haven't been able to try ConfigCheck out myself yet, but reports from a couple of trusted sources have suggested it's a fantastically useful tool, and you can't beat the price as it's FREE!

Tripwire® ConfigCheckTM is a free utility that rapidly assesses the security of VMware ESX 3.5 hypervisor configurations compared to the VMware Infrastructure 3 Security Hardening guidelines. Developed by Tripwire in cooperation with VMware, Tripwire ConfigCheck ensures ESX environments are properly configured—offering immediate insight into unintentional vulnerabilities in virtual environments—and provides the necessary steps towards full remediation when they are not.

If I have time next week, I plan to give this a whirl, but I'd suggest that if you've already implemented VMware or are planning to, you should make use of a utility such as this...until it's bundled into the platforms themselves ;)

Get your copy here.

Good move by Tripwire.

"Revolutionary" VirtSec Startup Emerges From Stealth

If Barracuda attempting to gobble up SourceFire today wasn't interesting enough, check this out...

WALTHAM, Mass., May 30 /PRNewswire/ -- Hyperbole, Inc., the the pioneer and leader in virtualization security solutions today announced it has emerged from stealth mode and raised $14 million in a Series A funding which it will use to expand its R&D efforts and grow its sales and distribution teams.

Hyperbole's flagship product, HyperTension, provides a zero footprint and forensically tight paradigm-shift in the emerging virtualization security (VirtSec) market by automatically protecting all virtual infrastructure against known or unknown attacks without the need for expensive and clumsy IDS, firewall and IPS technology. 

With no agent software and no hardware requirements save for a specially-constructed tamper-proof USB device called the HyperDrive, HyperTension is able to secure any virtualization platform automatically within seconds and with no downtime required.

HyperTension provides an undetectable ring compression insertion technology that injects itself into memory space transparently and utilizes the flash memory space available in PCI cards present in the system to load, thereby not corrupting the main heap and rendering itself undetectable. 

Further, HyperTension will probe for the presence of parallelized graphics processing units (GPU) from leading graphics card providers and if found, will utilize them to provide the compute cycles necessary for operation thereby not impacting the on-board main CPU or cache, further lessening the impact of the solution running in virtualized environments. 

This allows for massive computation capabilities used to provide real-time memory-space attack detection functionality which can be manually or automatically adjusted using our patented HyperSensitivity comb filter technology.

Hyperbole's patented HyperVentilation technology utilizes quantum cryptography and open source algorithms to create "holes" in memory to dynamically encrypt/decrypt the entire memory space of a virtualized host and upon register access, leverage commodity TPM solutions to authenticate and decrypt memory on the fly when used in conjunction with any of Hyperbole's partner-supplied whitelisting solutions.

Once accessed, HyperTension automatically performs an ASLR operation for pointer obfuscation and then re-encrypts the memory space using a newly-generated quantum key derived from the unique properties of the hashed cache entries from the rotating cipher.

This provides unbreakable security since only authorized applications can attempt to gain access to HyperVentilated memory space which is also encrypted to prevent unauthorized access.

...

Speechless. 

/Hoff

Pushing Virtual Buttons...

My last couple of VirtSec posts have caused quite a stir in certain circles.

The "debate" between who "owns" VirtSec that originated as part of my response to Simon Crosby of Citrix regarding the same has been picked up and amplified on multiple fronts.

Greg Ness from BlueLane wrote a piece referencing it that was cross-posted on virtualization.com and that even made its way up to VC/investment blogs such as seekingalpha.com (Citrix vs. Chris Hoff ;) and has had my mobile ringing/vibrating itself off my desk over the last week or so.

It's hard to believe sometimes just how many people -- and who -- reads my steaming pile of blogginess.

The second post of interest was in regard to the provenance of VMware's VMsafe and my reflection on prior art (Livewire) by VMware's Rosenblum & Garfinkel which seems as though it could be the progenitor of the upcoming technology.

The very tail-end update of that post referenced another piece of research produced by Komoku based upon similar work focused on rootkit defense. As I pointed out, Komoku was recently acquired by Microsoft.

I added those comments deliberately as a parenthetical -- almost like a bookmark -- because what I intended to do next was directly compare and contrast the technology architectures and approaches of VMware, Citrix and Microsoft as it relates to security integration.

It seems a bunch of really bright folks caught onto that because a slew of links (such as this one) followed -- driven mostly by Alessandro's (virtualization.info) post titled "Is Microsoft Working On VMsafe-like Framework"

I think that's an excellent question ;)

It's pretty clear where Citrix's CTO stands on the matter -- as flawed as I see his shortsighted market approach (note I didn't say *technical approach*) -- but Microsoft stands to gain an interesting foothold in regards to security should they play this game correctly.

I found it interesting that others are starting to recognize that the virtualization battle isn't going to be won by a shoot-out and the hypervisor-version of the OK corral. It's the effectiveness of the ecosystem and the ability for the channel to serve it up and the customers to implement it.

People are sick of sweeping up the decaying corpses of good technical solutions that suck in terms of integration, implementation, operationalization and accountable support -- especially when they have to keep paying for it. Ah the "best-in-breed" versus "good-enough" debate again?

Not to further pick on Citrix (or Xen specifically) but here's a great post from Schley Andrew Kutz from the searchservervirtualization.com blog titled "Xen: An endangered species in the virtualization ecosystem?":

While Citrix Systems’ Xen’s ubiquity may help the technology earn a legacy as the invisible hypervisor, it may also prove the most challenging next step for IT administrators and developers who want to find or develop software that leverages, supports or extends the Xen hypervisor.

...

While ultimately it may not prove difficult to develop cutting-edge technology compatible with the Xen hypervisor, it may prove so to market it. If you are in the business of selling virtualization add-on products, you want to ensure that your product is compatible with VMware Infrastructure, because that is where the sales are.

...

As Xen’s legacy may be to become the ubiquitous, embedded hypervisor for all to use, its strength may also be its greatest detriment to Xen-based virtualization platforms. Xen’s strength is its practical application as the invisible, reused, resold, embedded hypervisor, but invisibility just hasn’t worked in Citrix’s favor. Instead, it shields partners from building ecosystems around Xen and has marginalized the brand name.

Amen to that.

Take heed, Citrix. I maintain your CTO is blinded by what can only be described as a denial of market realities and an undying (arrogant) allegiance to what some might consider to be an architecturally superior product on some fronts, but a lacking solution on many others.

Securing the hypervisor is definitely important. However, securing both the hypervisor and the assets that sit on top of it by providing the most extensible, effective and manageable means of doing so is really what's important to customers. Sometimes, it has to be about more than where you came from. Sometimes it's about where you're going.

I'll be finishing up my post on where I think Microsoft ought to go shortly.

/Hoff

The Ghost Of Future's Past: VirtSec Innovation Circa 2002

One of the things I try to do when looking forward for inspiration in solving problems is to ensure that I spend enough time looking back to gain perspective.  I've been thinking a lot about models for virtualization security lately.

As I surveyed the options (or lack thereof) splayed about before me in terms of deployment options and available technology to solve some of the problems I've been researching, I was struck by what I can only describe as a ghost of future's past. 

It shouldn't really surprise me like it does, but I always giggle when reminded of my own favorite saying: "Security is like bellbottoms -- every 20 years or so, the same funny-looking kit comes back into style."

As it is with jeans, it is with security solutions.

I dredged up some of my collected research from moon's ago on the topic and dusted off a PDF that I had completely forgotten about as I was trying to piece together some vague semblance of something that strangely reminded me of VMware's VMsafe.

I cracked a gigantic smile when I saw the authors -- Tal Garfinkel and some guy named Mendel Rosenblum (now co-founder and chief scientist at VMware.)

The PDF in question is titled Virtual Machine Introspection ("productized" as LiveWire) and presents the following case:

In this paper we present a new architecture for building intrusion detection systems that provides good visibility into the state of the monitored host, while still providing strong isolation for the IDS, thus lending significant resistance to both evasion and attack.  

Our approach leverages virtual machine monitor (VMM) technology. This mechanism allows us to pull our IDS “outside” of the host it is monitoring, into a completely different hardware protection domain, providing a high-confidence barrier between the IDS and an attacker’s malicious code.

We achieve this through the use of a virtual machine monitor. Using this approach allows us to isolate the IDS from the monitored host but still retain excellent visibility into the host’s state. The VMM also offers us the unique ability to completely mediate interactions between the host software and the underlying hardware. We present a detailed study of our architecture, including Livewire, a prototype implementation. We demonstrate Livewire by implementing a suite of simple intrusion detection policies and using them to detect real attacks.

I got to thinking about the relevance of this approach because of some of the arguments that Simon Crosby made in our debate recently.  I wanted to spend some more time thinking about the architectural differences between VMware and Xen so I could try an appreciate the genesis of Simon's comments in context.

This paper and the Livewire prototype was created circa 2002.  It's six years later and we're just now starting to see products and technology being announced as "new and fresh"  that is basically just like Livewire.

While it's certainly not the first and only research on this topic, it's interesting to see that sometimes the wisdom of the past just takes just a little longer to cook before it's fully baked, ready for icing and ready to be consumed.

If VMsafe is an example of the evolution of prior art like Livewire, what else do we have to look forward to that's buried somewhere waiting to come back to life?  Oh wait, those mainframes are coming back, aren't they?  What's old is new again.

/Hoff

{Update: I also found some cool related stuff from Tim Fraser called Virtual Machine Introspection for Cognitive Immunity (kernel rootkit mitigation using VM Introspection) from Komoku which was acquired about a month ago by, gasp, Microsoft...}

Crosby: Xen and the Art of Marketcycle Maintenance

It seems I have fallen victim to a series of misunderstandings these days.

First there was Joanna-Gate and now Simon Crosby, Citrix's CTO, suggests in a blog entry titled "Chris Hoff & The Mother Of All Misunderstandings" that I'm puffing on the wrong end of my cigars for disagreeing with his position.

I'm a little concerned that Simon's response to me was issued on what is listed as the "beta" version of Citrix's official blog.  Perhaps the virtualized version hasn't made it out of QA yet? ;)

Simon's response was extremely well crafted to avoid responding to most of my actual points, was contextually oblique at points, and was a fantastic marketing piece for Xen Citrix, but I wish he'd paid more attention to the actual points within my post. 

Further his little quips/comments on his hyperlinks "Who is this guy, anyway?  Think before you type dude, we're not idiots," etc. didn't go unnoticed - cute but juvenile)

I am, however, honored that Simon would accord me the high-status of being "...normally fairly clued-in:"

I reckon that Hoff, who is normally fairly clued-in,  has put the smoking end of the cigar in his mouth before thinking through this argument. He's horribly confused, but as smug as always, so let me clarify what I said, and what it means.

...but I can assure you that I've only ever done that with a cigar once, and it was for a much better reason than blogging.  If you must know, it was Kentucky's finest bourbon.  That is all I'm going to say about that. 

I'm glad he's "clarifying" what he said, since I will also.  I seem to have that effect on people.  Must be the accent thing...

The reason for my allergic reaction to Simon's comments stem from my opinion that it is the responsibility of virtualization platform providers to ensure that their "[virtualized] data center operating system platforms of the future" don't become the next generation of insecure infrastructure.

Simon sums up his opinion:

In summary an assertion that the virtualization platform vendor has to fix the sad state of the OS/App world by making it secure is demanding too much.  It would mean that we have to be experts in every piece of system software including all of the vulnerabilities of all OSes and their apps.  In my view the reason the state of security is poor now is because of the monolithic approaches of traditional OS and app vendors. 

We will focus manically on our layer, make it secure, tiny and bulletproof to attack in its own right.  And we will work closely with experts in security of OSes and Apps to give them an opportunity to implement guest-level security outside the guest, through privileged interfaces that themselves are secure.

After 15 years of dealing with this crap, I respectfully suggest that it is not too much to ask and it's about time we stood up and did.  First  you criticize OS/App. vendors and blame them for the state of security because of their "monolithic approach" and then you go on to propose the exact same thing!

Focusing only on your little patch of grass is short-sighted and it won't work.  Just like it hasn't worked in the past.  It's a disaster waiting to happen, and you're enabling it. 

I shudder at the potential tunnel vision of virtualization platform providers only focusing on the security of the hypervisor without taking the bigger picture into consideration and expect a piecemeal approach to securing the expanse of the virtualized environment to suffice.

It's clear you're making arguments about security from an engineering and code-base perspective that is simply disconnected from the realities of what it means to actually deploy these solutions. 

Virtualization is more than just the hypervisor.  You should know that by now, Simon.  The company that acquired your company knows all about that.  The hypervisor will shortly become a commodity, so in the long term the value brought to bear has to be more than just an ultra-thin layer of code:

...and furthermore, we're going to deploy many of them:

I wish to make it clear that I hold all virtualization platform vendors to the same level of scrutiny and criticism, not just Citrix. 

I happen to like Xen very much.  I like VMware, also.  I think the latter is more realistic and measured when it comes to addressing the need and approach in recognizing that as a major layer in the infrastructure, there's more required than to just secure the hypervisor and leave the remaining mess to someone else to solve.

I think Simon's blog title is apropos, but I think the misunderstanding is his.

It's important to understand that I'm not suggesting that virtualization platform providers should secure the actual guest operating systems but they should enable an easier and more effective way of doing so when virtualized.

I mean that the virtualization platform providers should ensure the security of the instantiation of those guests as "hosted" by the virtualization platform.  In some cases this means leveraging technology present in the virtualization platform to do things that non-virtualized instances cannot. That's more than just securing the hypervisor.

Securing the hypervisor whilst closing your eyes to the likelihood that the majority of attacks against it and other guests will come from "guests" within the same system is planting your head in the sand.  That means that there will be a need to ensure that certain behaviors specific to the hosted guests are mitigated to ensure that bad things don't happen -- to the guest or the hypervisor.

Transferring the responsibility to secure the environment to third party security ISV's in order to secure the VM's and preventing them from compromising one another or the hypervisor is difficult for me to comprehend, especially when they are playing catch up of what virtualization means within the context of security.

Fundamentally, attempting to mate static and topology-dependent policies to incredibly dynamic and transitive technology delivered by virtualization will simply fail.  Third party security ISV's will simply require a complete re-tool to even get close to delivering this and will need to provide intimate hooks to allow for this policy/guest affinity to occur in the first place.

I consider the virtualization infrastructure layer as that of an operating system and as such, I would expect that the underpinning mechanicals are as sound and secure as possible while also ensuring that anything running on top of it is as secure as possible, also.

Let's take Microsoft (with or without Hyper-V) as an example:

Microsoft is fundamentally concerned now with making the OS as resilient and secure as possible whilst preventing the applications and interaction with elements riding on top of the OS from doing bad things to the system as a whole; this isn't just to protect the OS, but the assets on it. 

This is really what I'm getting at.  Yes, Microsoft is an OS provider.  Shortly, that OS provider will integrate virtualization directly into the operating system.  That means more, not less, direct integration and security embedded as a function of the virtualization platform.  Citrix, VMware, etc. are all just operating system vendors of a different shape and size.

It's unclear to me, Simon, whether your arguments are meant to justify a business model, a lack of planning, a crafty plan to perpetuate the security hamster wheel of pain, or all of the above.  It's clear to me, however, that you've not felt the pain of actually having to use the products you suggest should be deployed in order to secure this mess.

I promised myself I wouldn't turn this into one of those cut/paste blog pong entries, but the following really confused me:

But we are not in the business of specifically securing guests or their applications, other than through offering a secure virtualization platform.  Even VMware with VMsafe simply exposes APIs to third party security vendors, so that customers can choose their preferred security partner to secure guests.  I think that the VMware Determina acquisition was very smart, and that hints to me that VMware sees itself having a greater role in the security of guest OSes, since it could choose to be in the vulnerability checking business without 3rd party security vendors, but thus far they are working very openly with the ecosystem.

So which is it?  You've established that Citrix is not in the business of securing guests or applications (you must mean Xen specifically, because somebody at Citrix spent quite a bit of money on this stuff with their other acquisitions) and that you believe it to be a lousy idea, but you think that VMware's approach through their Determina acquisition as well as the capabilities of VMsafe is "...very smart?"

Simon, you're the CTO and I'm the security wonk.  If we didn't disagree, I'd be alarmed.  However, I think you might want to rethink your approach to how you market the security of your platform.

I've got a cigar for you anytime you want one.  I'll let you light it.

/Hoff

Citrix's Crosby & The Mother Of All Cop-Outs

In an article over at SearchSecurity.com, Simon Crosby, the CTO of Citrix, suggests that "Virtualization vendors [are] not in the security business." 

Besides summarizing what is plainly an obvious statement of fact regarding the general omission of integrated security (outside of securing the hypervisor) from most virtualization platforms, Crosby's statement simply underscores the woeful state we're in:

While virtualization vendors will do their role in protecting the hypervisor, they are not in the business of catching bad guys or discovering vulnerabilities, said Simon Crosby, chief technology officer of Citrix Systems.

Independent security vendors will play a critical role in protecting virtual environments, he said. "The industry has already decided a long time ago that third party vendors are required to secure any platform," Crosby said. In this interview, Crosby agrees that using virtual technology introduces new complexities and security issues.

He said the uncertainties will be addressed once the industry matures.

I'm sure it's reasonable to suggest that nobody expects virtualization platform providers to "...catch bad guys," but I do expect that they employ a significant amount of resources and follow an SDLC to discover vulnerabilities -- at least in their software.

Further, I don't expect that the hypervisor should be the place in which all security functionality is delivered, but simply transferring the lack of design and architecture forethought from the hypervisor provider to the consumer by expecting someone else to clean up the mess is just, well, typical.

I love the last line.  What a crock of shit.  We've seen how well this approach had worked with operating system vendors in the past, so why shouldn't the "next generation" of OS vendors -- virtualization platform providers -- follow suit and not provide for a secure operating environment?

Let's see, Microsoft is investing hugely in security.  Cisco is too.  Why would the other tip of the trident want to?  VMware's at least taking steps to deliver a secure hypervisor as well as API's to help secure the  VM's that run atop of it.   Where's Citrix in this...I mean besides late and complaining they weren't first?

So, in trade for the "open framework for security ecosystem partnership" cop-out, we get to wait for the self-perpetuating security industry hamster wheel of pain to come back full circle. 

The fact that the "industry" has "decided" that "third party vendors are required to secure any platform" simply points to the ignorance, arrogance and manifest destiny we endure at the hands of those who are responsible for the computing infrastructure we're all held hostage with. 

Just so I understand the premise, the security industry (or is it the virtualization industry?) has decided that the security industry instead of the OS/infrastructure (virtualization) vendors are the one's responsible to secure the infrastructure -- and thus our businesses!?  What a shocker.  Way to push for change, Simon.

I can't even describe how utterly pissed off these statements make me.

/Hoff

Virtualizing Security Will NOT Save You Money; It Will Cost You More

In my post titled "The Four Horsemen Of the Virtualization Apocalypse" I brought to light what I think are some nasty performance, resilience, configuration and capacity planning issues in regards to operationalizing virtualized security within the context of security solutions as virtual appliances/VM's in hosts.

This point was really intended to be discussed outside of the context of virtualizing security in physical switches, and I'll get to that point and what it means in relation to this topic in a later post.

I wanted to reiterate the point I made when describing the fourth horseman, Famine, summarized by what I called "Spinning VM straw into budgetary gold:"

By this point you probably recognize that you're going to be deploying the same old security  software/agents to each VM and then adding at least one VA to each physical host, and probably more.  Also, you're likely not going to do away with the hardware-based versions of these appliances on the physical networks.

That also means you're going to be adding additional monitoring points on the network and who is going to do that?  The network team?  The security team?  The, gulp, virtual server admin team?

What does this mean?  With all this consolidation, you're going to end up spending MORE on security in a virtualized world instead of less.

This is a really important issue because over the last few weeks, I've seen more and more discussions surrounding virtualization TCO and ROI calculations, but most simply do not take these points into consideration.

We talk about virtualization providing cooling, power and administrative cost-avoidance and savings.  We hear about operational efficiencies, improved service levels and agility, increased resource utilization and reduced carbon footprint. 

That's great, but with all this virtualized and converged functionality now "simplified" into a tab or two in the management console of your favorite virtualization platform provider, the complexity and operational issues related to security have just faded into the background and been thought of as having been absorbed or abstracted away.

I suppose that might point to why many simply think that security ought to be nothing more than a drop-down menu and checkbox because in most virtualization platforms, it is!

When thinking about this, I rationalized the experience and data points against my concern related to security's impact on performance, scale, and resiliency to arrive at what I think explains this behavior:

Most of the virtualization implementations today, regardless of whether they are client, server, production/QA or otherwise, are still internally-facing and internally-located.  There are not, based upon my briefings and research, a lot of externally-facing "classically DMZ'd" virtualized production instances.

This means that given the majority lack of segmentation of internal networks (from both a networking and security perspective,) the amount of network security controls in place are few.

Following that logic, one can then assume that short of the existing host-based controls which are put in place with every non-virtualized server install, most people continue this operational practice in their virtualized infrastructure; what they did yesterday is what they do today. 

Couple that with the lack of compelling security technologies available for deployment in the virtual hosts, most people have yet to start to implement multiple security virtual appliances on the same host.

Why would people worry about this now?   It's not really a problem...now.

When we start to see folks ramp up virtual host-based security solutions to protect against intra-vm threats and vulnerabilities (whether internally or externally-facing) as well as to prevent jail-breaking and leapfrog attacks against the underlying hypervisors, we'll start to see these problems bubble to the surface.

What are your thoughts?  Are you thinking about these issues as you plan your virtualization roll-outs?

/Hoff

The Five Laws Of Virtualization - Not Immutable Any More?

Update: Please read the comments section.  Rather than force playing blog pong, I've cross-posted some of the comment thread from Lindstrom's blog.

I believe I've offered up a clear present and future case that invalidates "immutable" law #1. Pete, of course, disagrees...

--

I've commented a couple of times about the confusingly contradictory nature of Lindstrom's Burton's "Five Immutable Laws of Virtualization."  I go back every once and a while and try to utilize them as suggested by their author to see what pops out the other end:

When combining the standard risk principles with an understanding of the use cases of virtualization, a set of immutable laws can be derived to assist in securing virtual environments

I'm not sure I really ever got an answer to what those "...standard risk principles" are and as such, there seems to exist a variability based upon interpretation that again makes me scratch my head when staring at the word "immutable."

So I try and overlook the word (as did the author/editor in the title of the Baseline magazine article below -- it was omitted) and I find myself back where I started which sort of makes sense given the somewhat reflexive and corollary nature of these "laws."   

This is where I get stuck.  I don't know whether to interpret each law as though it can stand on its own or the group as a whole.

Basically, I have a hard time seeing how they enable making more effective risk management decisions any easier.  I will admit, it could just be me...

Further, I've noticed the very careful choice of words used in these laws, and interestingly they don't appear to be consistently referenced which would defeat the purpose of calling them "immutable," no?

Take for example the original wording of the five laws from Burton's original minting and compare it against an article appearing in Baseline magazine from the same author(s) -- Lindstrom in this case:

Original Burton Article Example:

Law 1: Attacks against the OS and applications of a physical system have the exact same damage potential against a duplicate virtual system.

Baseline Magazine Article Example:

Law 1. Attacking a virtual combination of operating systems and applications is exactly the same as attacking the physical system it replicates.

This example may seem subtle and unimportant, but I maintain it is not.  I suggest that they mean very different things indeed.  I mean, if these are "laws," they're not something you get to reword at a whim.  I trust I don't have to  explain why.

One could have lots of fun with the Constitution if that were the case. ;)

There are additional differences scattered throughout the two articles.  See if they appeal differently to you as they did to me.

Now, I'm sure Pete's going to suggest I'm picking nits and that I'm missing the spirit and intent of these "laws," but before he does, I'm going to remind him that I didn't come up with the title, he did.  I'm merely stuck on trying to assess whether these are actually "immutable" or "refutable" but I am admittedly still having trouble getting past step #1.

Help a brother out.  Explain these to me to where they make sense.  Pete tried and it didn't stick.  Maybe you can help?

/Hoff

Poetic Virtual Security

I was at Starbucks with my four year old.  She was laying down the Dr. Seuss
with aplomb so I was inspired to dig deep and show her how the old man can
ebb and flow.

I swear to $diety that upon hearing this she rolled her eyes and said something like "Dad, you had me at 'virtualization.' "  At that point she quickly pointed to my iPhone and asked if I would purchase the latest Hannah Montana song on iTunes...<sigh>

You can see more of my poetic ramblings here (scroll down after the jump.)

---

When debating the future of secure virtualization
It's wise to reflect on its very creation

Some say poor code is the reason it's here
while others use doubt and (un)certainty's fear

Economically speaking the V-word's a boon
operationally, though, it showed up too soon

Duties, once separate, are now all a-blended
one moat, lots of castles -- the model's up-ended

Competency and skillsets come into play
Who owns the stack?  Well, that's hard to say

Can an admin whose mad skillz focus on the OS,
really be trusted to manage this mess?

The virtual sysadmin owns the keys to the kingdom
but it's hard to fix hosts when you can't even ping 'dem!

Operational silos have now become worse
since the virtual admins control all the purse

The network and security wonks try to fudge it
but switches and firewalls just don't get budget

Security, network, storage, and host
if you push the wrong button it all becomes toast

Our current security solutions don't cope
but the dealers keep pushing their VirtSec straight dope

I don't want to come off like a VirtSec despiser,
but to protect our crown jewels it's all HYPErvisor

Don't worry my friends, no need to be scared
your whole infrastructure will be VMware'd

...or Xen'd, or sPath'd or perhaps Hyper-V'd
virtualization, I'm told, will solve everyone's need

Organizational issues are really what matter
there's no real need to make our vendors much fatter

Focus first on improving your present situation
like assessing your risk and host segmentation

Get a grip on the basics and work up from there
don't give into the hype, doubt, confusion or fear

That's it boys and girls till I rhyme once again
Stay happy, stay secure, and now...

EOM

All Your Virtualized PCI Compliance Are Belong To Us...

Another interesting example I use in my VirtSec presentations when discussing the challenges of what I describe as Phase 2 of virtualization -- virtualizing critical applications and things like Internet-facing infrastructure in DMZ's -- is the notion of compliance failures based on existing and upcoming revisions to regulatory requirements.

Specifically, I use PCI/DSS to illustrate that in many cases were one to take a highly-segmented and stratified "defense-in-depth" architecture that is today "PCI compliant" and virtualize it given presently available options, you'd likely find yourself out of compliance given the current state of technology solutions and auditing standards used to assess against.

Then again, you might just pass with flying colors while being totally insecure.

Here's a fantastic example from Eric Siebert over at the TechTarget Virtualization blog.  Check this out, it's a doozie!

Having just survived another annual PCI compliance audit, I was again surprised that the strict standards for securing servers that must be followed contain nothing specific concerning virtual hosts and networks. Our auditor focused on guest virtual machines (VMs), ensuring they had up-to-date patches, locked-down security settings and current anti-virus definitions. But ironically, the host server that the virtual machines were running on went completely ignored. If the host server was compromised, it wouldn’t matter how secure the VMs were because they could be easily accessed. Host servers should always be securely locked down to protect the VMs which are running on them.

It seems that much of the IT industry has yet to react to the virtualization trend, having been slow in changing procedures to adjust to some of the unconventional concepts that virtualization introduces. When I told our auditor that the servers were virtual, the only thing he wanted to see was some documentation stating that the remote console sessions to the VMs were secure. It’s probably just a matter of time before specific requirements for virtual servers are introduced. In fact, a recent webinar takes up this issue of whether or not virtualized servers can be considered compliant, addressing section 2.2.1 of the PCI DSS which states, “Implement only one primary function per server”; that is to say, web servers, database servers and DNS should be implemented on separate servers. Virtual servers typically have many functions running on a single physical server, which would make them noncompliant.

So let's assume that what Eric talks about in section 2.2.1 of PCI/DSS holds true, that basically means two things: (1) PCI/DSS intimates that virtualization cannot provide the same level of security as non-virtualized infrastructure and (2) you won't be able to virtualize infrastructure governed by PCI/DSS if you expect to be compliant.

Now, this goes toward the stuff Mogull and I were talking about in terms of assessing risk and using the notion of "zone defense" for asset segmentation in virtualized infrastructure. 

Here's a snippet from my VirtSec preso on the point:

Further, as I mentioned in my post titled "Risky Business -- The Next Audit Cycle: Bellweather Test for Critical Production Virtualized Infrastructure," this next audit cycle is going to be interesting for many companies...

Yippeee!

/Hoff

Clouding the Issue: Separating "Securing Virtualization" from "Virtualizing Security"

My goal in the next couple of posts is to paint some little vignettes highlighting some of the more interesting points I raise in my presentation series "Virtualization: Floor Wax, Dessert Topping and the End Of Information Security As We Know It."

The first issue up for discussion is the need to recognize and separate two concerns which are unfortunately most often intertwined when companies are considering virtualization and its impact to their IT operations and security programs. 

My goal here is not to try and explain away every nuance of this slide or push a conclusion on anybody, but instead plant the seeds and set the premise for discussion's sake.

The slide to the left sums up the point reasonably well, but here's the associated scaled-down narrative that accompanies this slide:

Companies need to approach addressing each of these issues by assessing the risk associated with each separately and then juxtaposed.

Treating them as a single concern -- as most do -- leads to an unfortunate series of chicken-egg debates that usually do not address the things that really matter in the first place.

The point here is that while these concerns are very much related and both important, the order in which they are addressed is often critical.

Specifically, one can take an incredibly secure solution and yet still manage to deploy it in an incredibly insecure manner.  Even if the virtualization platform one chooses is (by some mythical standard) impervious to compromise (*cough*,) given specific configuration constraints, deviations from those constraints can lead to exposure.

If the manner in which virtualization platforms are configured, managed, monitored and secured after you've already deployed them are not consistent with the rigor and diligence we've applied to our non-virtualized infrastructure (and by observation they are not,) worrying about how secure or insecure your VMM platforms are is a waste of synaptic processes.

My experience has shown that most organizations have simply plowed ahead and accepted or ignored the risk associated with deploying virtualization platforms, accepting on blind faith the claims of virtualization vendors and assuming that the VMM providing the abstraction layer between hardware and software is at least as secure (if not more so) as a non-virtualized installation of the operating system.

This is usually done because the economic benefits of virtualization which are absolutely quantifiable far outweigh the perceived risks associated with virtualization which are not (or are at least difficult to produce.)

I'm unsure how exactly most companies are assessing risk against their virtualized environments formally since many of them admit to not having a risk assessment methodology in place to do so.

It would seem that most folks simply look at the known vulnerabilities associated with a vendor's VMM and the current threatscape and make a swag as to the resultant residual risk given any compensating controls that might be in place.  In many cases, however, the "risk" we're debating is based upon threats and vulnerabilities that may not even exist, so we're academically making judgment calls based on possibility versus probability.

Yikes.

How many times have you entered into debate with *someone* in IT, security, audit or the business arguing about "securing virtualization" after someone's seen a "Blue Pill" presentation when in all honestly the company has already deployed hundreds of VM's and still hasn't segmented the network or built a risk assessment framework to quantify the business impact?

See what I mean?

/Hoff

Ghost In the Machine: IBM's New "Phantom" VirtSec Solution (?)

I had another post-RSA press release show up in my mailbox today from IBM again pitching their "...breakthrough research initiative from IBM X-Force and IBM Research, code-named "Phantom", which offers businesses a new means of securing virtualized server environments."

Besides the rumblings at RSA, I haven't been briefed on this as of yet, but let's explore what we have thus far, keeping it mind that this is described as an "initiative" and not a "product:"

At Phantom's core is industry-leading network and host intrusion protection used to guard the virtual environment and the machines from the inside out. The new technology sits in a secure, isolated partition and integrates with the hypervisor - the layer of management software that coordinates calls between operating systems and computer hardware.

In this description, Phantom is confusingly fra