Rational Survivability

Crosby: Xen and the Art of Marketcycle Maintenance

It seems I have fallen victim to a series of misunderstandings these days.

First there was Joanna-Gate and now Simon Crosby, Citrix's CTO, suggests in a blog entry titled "Chris Hoff & The Mother Of All Misunderstandings" that I'm puffing on the wrong end of my cigars for disagreeing with his position.

I'm a little concerned that Simon's response to me was issued on what is listed as the "beta" version of Citrix's official blog.  Perhaps the virtualized version hasn't made it out of QA yet? ;)

Simon's response was extremely well crafted to avoid responding to most of my actual points, was contextually oblique at points, and was a fantastic marketing piece for Xen Citrix, but I wish he'd paid more attention to the actual points within my post. 

Further his little quips/comments on his hyperlinks "Who is this guy, anyway?  Think before you type dude, we're not idiots," etc. didn't go unnoticed - cute but juvenile)

I am, however, honored that Simon would accord me the high-status of being "...normally fairly clued-in:"

I reckon that Hoff, who is normally fairly clued-in,  has put the smoking end of the cigar in his mouth before thinking through this argument. He's horribly confused, but as smug as always, so let me clarify what I said, and what it means.

...but I can assure you that I've only ever done that with a cigar once, and it was for a much better reason than blogging.  If you must know, it was Kentucky's finest bourbon.  That is all I'm going to say about that. 

I'm glad he's "clarifying" what he said, since I will also.  I seem to have that effect on people.  Must be the accent thing...

The reason for my allergic reaction to Simon's comments stem from my opinion that it is the responsibility of virtualization platform providers to ensure that their "[virtualized] data center operating system platforms of the future" don't become the next generation of insecure infrastructure.

Simon sums up his opinion:

In summary an assertion that the virtualization platform vendor has to fix the sad state of the OS/App world by making it secure is demanding too much.  It would mean that we have to be experts in every piece of system software including all of the vulnerabilities of all OSes and their apps.  In my view the reason the state of security is poor now is because of the monolithic approaches of traditional OS and app vendors. 

We will focus manically on our layer, make it secure, tiny and bulletproof to attack in its own right.  And we will work closely with experts in security of OSes and Apps to give them an opportunity to implement guest-level security outside the guest, through privileged interfaces that themselves are secure.

After 15 years of dealing with this crap, I respectfully suggest that it is not too much to ask and it's about time we stood up and did.  First  you criticize OS/App. vendors and blame them for the state of security because of their "monolithic approach" and then you go on to propose the exact same thing!

Focusing only on your little patch of grass is short-sighted and it won't work.  Just like it hasn't worked in the past.  It's a disaster waiting to happen, and you're enabling it. 

I shudder at the potential tunnel vision of virtualization platform providers only focusing on the security of the hypervisor without taking the bigger picture into consideration and expect a piecemeal approach to securing the expanse of the virtualized environment to suffice.

It's clear you're making arguments about security from an engineering and code-base perspective that is simply disconnected from the realities of what it means to actually deploy these solutions. 

Virtualization is more than just the hypervisor.  You should know that by now, Simon.  The company that acquired your company knows all about that.  The hypervisor will shortly become a commodity, so in the long term the value brought to bear has to be more than just an ultra-thin layer of code:

...and furthermore, we're going to deploy many of them:

I wish to make it clear that I hold all virtualization platform vendors to the same level of scrutiny and criticism, not just Citrix. 

I happen to like Xen very much.  I like VMware, also.  I think the latter is more realistic and measured when it comes to addressing the need and approach in recognizing that as a major layer in the infrastructure, there's more required than to just secure the hypervisor and leave the remaining mess to someone else to solve.

I think Simon's blog title is apropos, but I think the misunderstanding is his.

It's important to understand that I'm not suggesting that virtualization platform providers should secure the actual guest operating systems but they should enable an easier and more effective way of doing so when virtualized.

I mean that the virtualization platform providers should ensure the security of the instantiation of those guests as "hosted" by the virtualization platform.  In some cases this means leveraging technology present in the virtualization platform to do things that non-virtualized instances cannot. That's more than just securing the hypervisor.

Securing the hypervisor whilst closing your eyes to the likelihood that the majority of attacks against it and other guests will come from "guests" within the same system is planting your head in the sand.  That means that there will be a need to ensure that certain behaviors specific to the hosted guests are mitigated to ensure that bad things don't happen -- to the guest or the hypervisor.

Transferring the responsibility to secure the environment to third party security ISV's in order to secure the VM's and preventing them from compromising one another or the hypervisor is difficult for me to comprehend, especially when they are playing catch up of what virtualization means within the context of security.

Fundamentally, attempting to mate static and topology-dependent policies to incredibly dynamic and transitive technology delivered by virtualization will simply fail.  Third party security ISV's will simply require a complete re-tool to even get close to delivering this and will need to provide intimate hooks to allow for this policy/guest affinity to occur in the first place.

I consider the virtualization infrastructure layer as that of an operating system and as such, I would expect that the underpinning mechanicals are as sound and secure as possible while also ensuring that anything running on top of it is as secure as possible, also.

Let's take Microsoft (with or without Hyper-V) as an example:

Microsoft is fundamentally concerned now with making the OS as resilient and secure as possible whilst preventing the applications and interaction with elements riding on top of the OS from doing bad things to the system as a whole; this isn't just to protect the OS, but the assets on it. 

This is really what I'm getting at.  Yes, Microsoft is an OS provider.  Shortly, that OS provider will integrate virtualization directly into the operating system.  That means more, not less, direct integration and security embedded as a function of the virtualization platform.  Citrix, VMware, etc. are all just operating system vendors of a different shape and size.

It's unclear to me, Simon, whether your arguments are meant to justify a business model, a lack of planning, a crafty plan to perpetuate the security hamster wheel of pain, or all of the above.  It's clear to me, however, that you've not felt the pain of actually having to use the products you suggest should be deployed in order to secure this mess.

I promised myself I wouldn't turn this into one of those cut/paste blog pong entries, but the following really confused me:

But we are not in the business of specifically securing guests or their applications, other than through offering a secure virtualization platform.  Even VMware with VMsafe simply exposes APIs to third party security vendors, so that customers can choose their preferred security partner to secure guests.  I think that the VMware Determina acquisition was very smart, and that hints to me that VMware sees itself having a greater role in the security of guest OSes, since it could choose to be in the vulnerability checking business without 3rd party security vendors, but thus far they are working very openly with the ecosystem.

So which is it?  You've established that Citrix is not in the business of securing guests or applications (you must mean Xen specifically, because somebody at Citrix spent quite a bit of money on this stuff with their other acquisitions) and that you believe it to be a lousy idea, but you think that VMware's approach through their Determina acquisition as well as the capabilities of VMsafe is "...very smart?"

Simon, you're the CTO and I'm the security wonk.  If we didn't disagree, I'd be alarmed.  However, I think you might want to rethink your approach to how you market the security of your platform.

I've got a cigar for you anytime you want one.  I'll let you light it.

/Hoff

Hypervisors Are Becoming a Commodity...Virtualization Is a Feature?

A couple of weeks ago I penned a blog entry titled "The Battle for the HyperVisor Heats Up" in which I highlighted an announcement from Phoenix Technologies detailing their entry into the virtualization space with their BIOS-enabled VMM/Hypervisor offering called HyperCore.

This drew immediate parallels (no pun intended) to VMware and Xen's plans to embed virtualization capabilities into hardware.

The marketing continues this week with interesting announcements from Microsoft, Oracle and VMware:

1. VMware offers VMware Server 2 as a free virtualization product to do battle against...
2. Oracle offering "Oracle VM" for free (with paid support if you like) which claims to be 3 times as efficient than VMWare -- based on Xen.
3. Microsoft officially re-badged its server virtualization technology as Hyper-V (nee Veridian)
   detailing both a stand-alone Hyper-V Server as well technology integrated into W2K8 Server.
   

It seems that everyone and their mother is introducing a virtualization platform and the underpinning of commonality between basic functionality demonstrates how the underlying virtualization enabler -- the VMM/Hypervisor -- is becoming a commodity.

We are sure to see fatter, thinner, faster, "more secure" or more open Hypervisors, but this will be an area with less and less differentiation.  Table stakes.  Everything's becoming virtualized, so a VMM/Hypervisor will be the underlying "OS" enabling that transformation.

To illustrate the commoditization trend as well as a rather fractured landscape of strategies, one need only look at the diversity in existing and emerging VMM/Hypervisor solutions.   Virtualization strategies are beginning to revolve around a set of distinct approaches where virtualization is:

1. Provided for and/or enhanced in hardware (Intel, AMD, Phoenix)
2. A function of the operating system (Linux, Unix, Microsoft)
3. Delivered by means of an enabling software layer (nee platform) that is deployed across your entire infrastructure (VMware, Oracle)
4. Integrated into the larger Data Center "Fabric" or Data Center OS (Cisco)
5. Transformed into a Grid/Utility Computing model for service delivery

The challenge for a customer is making the decision on whom to invest it now.  Given the fact that there is not a widely-adopted common format for VM standardization, the choice today of a virtualization vendor (or vendors) could profoundly affect one's business in the future since we're talking about a fundamental shift in how your "centers of data" manifest.

What is so very interesting is that if we accept virtualization as a feature defined as an abstracted platform isolating software from hardware then the next major shift is the extensibility, manageability and flexibility of the solution offering as well as how partnerships knit out between the "platform" providers and the purveyors of toolsets.

It's clear that VMware's lead in the virtualization market is right inline with how I described the need for differentiation and extensibility both internally and via partnerships. 

VMotion is a classic example; it's clearly an internally-generated killer app. that the other players do not currently have and really speaks to being able to integrate virtualization as a "feature" into the combined fabric of the data center.  Binding networking, storage, computing together is critical.  VMware has a slew of partnerships (and potential acquisitions) that enable even greater utility from their products.

Cisco has already invested in VMware and a recent demo I got of Cisco's VFrame solution shows they are serious about being able to design, provision, deploy, secure and manage virtualized infrastructure up and down the stack, including servers, networking, storage, business process and logic.

In the next 12 months or so, you'll be able to buy a Dell or HP server using Intel or AMD virtualization-enabled chipsets pre-loaded with multiple VMM/Hypervisors in either flash or BIOS.  How you manage, integrate and secure it with the rest of your infrastructure -- well, that's the fun part, isn't it?

I'll bet we'll see more and more "free" commoditized virtualization platforms with the wallet ding coming from the support and licenses to enable third party feature integration and toolsets.

/Hoff

Take5 (Episode #5) - Five Questions for Allwyn Sequeira, SVP of Product Operations, Blue Lane

This fifth episode of Take5 interviews Allwyn Sequeira, SVP of Product Operations for Blue Lane.  

First a little background on the victim:

Mr. Palmer is the President and Chief Executive Officer of Blue Lane Technologies. Previously, Jeff served as President of GetThere, the leading online corporate travel procurement solution, based in Menlo Park, CA. While at the company, GetThere grew to become the largest factor in the corporate online travel market, holding 70% market share among the top 100 US corporations, #1 share in the overall market with more than 3000 customers worldwide, and transaction volumes exceeding one million per month. GetThere went public on the NASDAQ in 1999 and was acquired by Sabre Holdings (NYSE: TSG) for $750 million in 2000.
         
Earlier in his career, Mr. Palmer served in a variety of general management and marketing officer roles at Tristrata Security, Memco Software, Pilot Software and BBN, establishing experience in enterprise networking, security, and applications software.
         
Mr. Palmer holds a Bachelor of Science degree from the Massachusetts Institute of Technology and an MBA from Harvard Business School.

Allwyn Sequeira is Senior Vice President of Product Operations at Blue Lane Technologies, responsible for managing the overall product life cycle, from concept through research, development and test, to delivery and support. He was previously the Senior Vice President of Technology and Operations at netVmg, an intelligent route control company acquired by InterNap in 2003, where he was responsible for the architecture, development and deployment of the industry-leading flow control platform. Prior to netVmg, he was founder, Chief Technology Officer and Executive Vice President of Products and Operations at First Virtual Corporation (FVC), a multi-service networking company that had a successful IPO in 1998. Prior to FVC, he was Director of the Network Management Business Unit at Ungermann-Bass, the first independent local area network company. Mr. Sequeira has previously served as a Director on the boards of FVC and netVmg.

Mr. Sequeira started his career as a software developer at HP in the Information Networks Division, working on the development of TCP/IP protocols. During the early 1980's, he worked on the CSNET project, an early realization of the Internet concept. Mr. Sequeira is a recognized expert in data networking, with twenty five years of experience in the industry, and has been a featured speaker at industry leading forums like Networld+Interop, Next Generation Networks, ISP Con and RSA Conference.

Mr. Sequeira holds a Bachelor of Technology degree in Computer Science from the Indian Institute of Technology, Bombay, and a Master of Science in Computer Science from the University of Wisconsin, Madison.   

Allwyn, despite all this good schoolin' forgot to send me a picture, so he gets what he deserves ;)
(Ed: Yes, those of you quick enough were smart enough to detect that the previous picture was of Brad Pitt and not Allwyn.  I apologize for the unnecessary froth-factor.)

 Questions:

1) Blue Lane has two distinct product lines, VirtualShield and PatchPoint.  The former is a software-based solution which provides protection for VMware Infrastructure 3 virtual servers as an ESX VM plug-in whilst the latter offers a network appliance-based solution for physical servers.  How are these products different than either virtual switch IPS' like Virtual Iron or in-line network-based IPS's?

IPS technologies have been charged with the incredible mission of trying to protect everything from anything.  Overall they've done well, considering how much the perimeter of the network has changed and how sophisticated hackers have become. Much of their core technology, however, was relevant and useful when hackers could be easily identified by their signatures. As many have proclaimed, those days are coming to an end.

A defense department official recently quipped, "If you offer the same protection for your toothbrushes and your diamonds you are bound to lose fewer toothbrushes and more diamonds."  We think that data center security similarly demands specialized solutions.  The concept of an enterprise network has become so ambiguous when it comes to endpoints and devices and supply chain partners, etc. we think its time to think more realistically in terms of trusted, yet highly available zones within the data center.

It seems clear at this point that different parts of the network need very different security capabilities.  Servers, for example need highly accurate solutions that do not block or impede good traffic and can correct bad traffic, especially when it comes to closing network-facing vulnerability windows.  They need to maintain availability with minimal latency for starters; and that has been a sort of Achilles heel for signature-based approaches.  Of course, signatures also bring considerable management burdens over and beyond their security capabilities.

No one is advocating turning off the IPS, but rather approaching servers with more specialized capabilities.  We started focusing on servers years ago and established very sophisticated application and protocol intelligence, which has allowed us to correct traffic inline without the noise, suspense and delay that general purpose network security appliance users have come to expect.

IPS solutions depend on deep packet inspection typically at the perimeter based on regexp pattern matching for exploits.  Emerging challenges with this approach have made alert and block modes absolutely necessary as most IPS solutions aren't accurate enough to be trusted in full library block. 

Blue Lane uses a vastly different approach.  We call it deep flow inspection/correction for known server vulnerabilities based on stateful decoding up to layer 7.  We can alert, block and correct, but most of are deployments are in correct mode, with our full capabilities enabled. From an operational standpoint we have substantially different impacts.

A typical IPS may have 10K signatures while experts recommend turning on just a few hundred.  That kind of marketing shell game (find out what really works) means that there will be plenty of false alarms, false positives and negatives and plenty of tuning.  With polymorphic attacks signature libraries can increase exponentially while not delivering meaningful improvements in protection. 

Blue Lane supports about 1000 inline security patches across dozens of very specific server vulnerabilities, applications and operating systems.  We generate very few false alarms and minimal latency.  We don't require ANY tuning.  Our customers run our solution in automated, correct mode.

The traditional static signature IPS category has evolved into an ASIC war between some very capable players for the reasons we just discussed.Exploding variations of exploits and vectors means that exploit-centric approaches will require more processing power.

Virtualization is pulling the data center into an entirely different direction, driven by commodity processors.  So of course our VirtualShield solution was a much cleaner setup with a hypervisor; we can plug into the hypervisor layer and run on top of existing hardware, again with minimal latency and footprint.

You don't have to be a Metasploit genius to evade IPS signatures.  Our higher layer 7 stateful decoding is much more resilient. 

2) With zero-days on the rise, pay-for-play vulnerability research and now Zero-Bay (WabiSabiLabi) vulnerability auctions and the like, do you see an uptake in customer demand for vulnerability shielding solutions?

Exploit-signature technologies are meaningless in the face of evanescent, polymorphic threats, resulting in 0-day exploits. Slight modifications to signatures can bypass IPSes, even against known vulnerabilities.  Blue Lane technology provides 0-day protection for any variant of an exploit against known vulnerabilities.  No technology can provide ultimate protection against 0-day exploits based on 0-day vulnerabilities. However, this requires a different class of hacker.

3) As large companies start to put their virtualization strategies in play, how do you see customers addressing securing their virtualized infrastructure?  Do they try to adapt existing layered security methodologies and where do these fall down in a virtualized world?

I've explored this topic in depth at the Next Generation Data Center conference last week. Also, your readers might be interested in listening to a recent podcast: The Myths and Realities of Virtualization Security: An Interview. 

To summarize, there are a few things that change with virtualization, that folks need to be aware of.  It represents a new architecture.  The hypervisor layer represents the un-tethering and clustering of VMs, and centralized control.  It introduces a new virtual network layer.  There are entirely new states of servers, not anticipated by traditional static security approaches (like instant create, destroy, clone, suspend, snapshot and revert to snapshot). 

Then you'll see unprecedented levels of mobility and new virtual appliances and black boxing of complex stacks including embedded databases.  Organizations will have to work out who is responsible for securing this very fluid environment.  We'll also see unprecedented scalability with Infiniband cores attaching LAN/SAN out to 100's of ESX hypervisors and thousands of VMs.

Organizations will need the capability to shield these complex, fluid environments; because trying to keep track of individual VMs, states, patch levels, locations will make tuning an IPS for polymorphic attacks look like childs play in comparison.   Effective solutions will need to be highly accurate, low latency solutions deployed in correct mode. Gone will be the days of man-to-man blocking and tuning.  Here to stay are the days of zone defense.

4) VMware just purchased Determina and intends to integrate their memory firewall IPS product as an ESX VM plug-in.  Given your early partnership with VMware, are you surprised by this move?  Doesn't this directly compete with the VirtualSheild offering?

I wouldn't read too much into this. Determina hit the wall on sales, primarily because it's original memory wall technology was too intrusive, and fell short of handling new vulnerabilities/exploits.

This necessitated the LiveShield product, which required ongoing updates, destroying the value proposition of not having to touch servers, once installed. So, this is a technology/people acquisition, not a product line/customer-base acquisition.

VMware was smart to get a very bright set of folks, with deep memory/paging/OS, and a core technology that would do well to be integrated into the hypervisor for the purpose of hypervisor hardening, and interVM isolation. I don't see VMware entering the security content business soon (A/V, vulnerabilities, etc.). I see Blue Lane's VirtualShield technology integrated into the virtual networking layer (vSwitch), as a perfect complement to anything that will come out of the Determina acquisition.

5) Citrix just acquired XenSource.  Do you have plans to offer VirtualShield for Xen? 

A smart move on Citrix's part to get back into the game. Temporary market caps don't matter. Virtualization matters. If Citrix can make this a two or three horse race, it will keep the VMware, Citrix, Microsoft triumvirate on their toes, delivering better products, and net good for the customer.

Regarding BlueLane, and Citrix/Xensource, we will continue to pay attention to what customers are buying as they virtualize their data centers. For now, this is a one horse show :-)

Citrix Buying XenSource -- It's About Time(ing)

This will be short and sweet.  Citrix's announcement that they will clip a swell $500 Million to acquire XenSource on the tail of VMware's IPO makes nothing but sense.  The timing is interesting; waiting for VMware's IPO both validated the move but one has to wonder if it jacked the price any.

I can't wait to see how this maps out over time across Citrix's product lines which are still fairly siloed at this point.  Leveraging XenSource's technology is a force multiplier across many elements of their offerings. It's clear what the first moves will be, but I'm really interested in the longer term play.

At any rate, this is a fantastic strategic move for Citrix; these guys are poised to continue their march to take on Cisco as they become a robust platform for application and content delivery.*   If you take a look at their M&A activity over the last few years, it's on a direct collision course with Cisco in many vectors. 

The big difference is, you can bolt their solution on instead of having to bake it in and these guys already have a footprint and expertise in the server and client consolidation markets.

Orthogonally, I wonder what effect this might have on f5?  Any thoughts there?

Then there's Microsoft.  This may be a huge opportunity for other players such as SWsoft  to reinforce defensive positioning by shoring up relationships that otherwise might have gone XS's way.

It's going to get messy boys and girls.

This acquisition certainly has its challenges, but it really positions Citrix with as a complement to their existing product offerings.

/Hoff

*It gets more interesting strategically from a defensive position given Cisco's recent investment of $150M in VMware prior to their IPO and my commentary on the matter here.