Rational Survivability

On the Overcast Podcast with Geva Perry and James Urquhart

Geva and James were kind (foolish?) enough to invite me onto their Overcast podcast today:

In this podcast we talk to Christopher Hoff, renowned information security expert, and especially security in the context of virtualization and cloud computing. Chris is the author of the Rational Survivability blog, and can be followed as @Beaker on Twitter.
Show Notes:

• Chris talks about some of the myths and misconceptions about security in the cloud. He addresses the claim that Cloud Providers Are Better At Securing Your Data Than You Are and the benefits and shortcomings of security in the cloud.
• We talk about Chris's Taxonomy of Cloud Computing (excuse me, model of cloud computing)
• Chris goes through some specific challenges and solutions for PCI-compliance in the cloud
• Chris examines some of the security issues associated with multi-tenant architecture and virtualization

Check it out here.

/Hoff 

Oh Noes: We Can't Monitor/Protect Against Intra-VM Traffic!

I got a press release in my inbox this morning that made me cringe.  It came from a vendor who produces a "purpose-built virtual firewall."

The press release details a customer case study that I found typical of how security solutions are being marketed in the virtualization space today, which again is really more about visibility than pure "security" and preys mostly on poor planning and fundamental issues stemming from treating "security" like a band-aid instead of an element of enterprise architecture.

When we start to cross the streams as to the realities of virtualization, the security implications thereof and making promises to solve problems with products which may or may not be deserving of investment given an assessment of risk, especially in today's trying economic climate, it makes me cranky.

I'm just tiring of the mixing of metaphors in the marketing of these "solutions."

I was specifically annoyed by a couple of statements in the press release and since I haven't had my coffee, I thought I'd point out a few to further underscore what I present in my Four Horsemen presentation regarding where we are in the solution continuum today.

To wit:

[Customer] has selected the [Vendor's] virtual firewall to secure its virtual environment and mitigate an attack before it could hit their network. 

Given the fact that to get to a VM you generally have to (1) utilize the physical network and (2) transit the vSwitch in the VMM, the reality is that an attack has already "hit their network" long before it gets to the VM or the virtual firewall, at least given today's available offerings.  There is no magic security fairy dust that will mitigate an attack presciently.

If you put VM's into production that are already infected, you have other problems to solve...

“After moving our production applications to a virtualized environment we realized that we lacked security; I had no visibility into what was going on between VMs and a virtual attack could take down our network,” said [Customer.]  “We sought the same level of security for our virtual environment that we had with our physical network.”

This indicates a lack of proper risk management and planning on the part of the [Customer.]  Further, it underscores an example I use in the Four Horsemen which concerns which tools in a multi-server physical deployment did the [Customer] use to monitor/protect in-line traffic between these physical machines? 

The {Customer] must have done this since the press release suggests they demand the "...same level of security for [their] virtual environment that [they] had with [their] physical network."

Did the [Customer] have each physical server on it's own VLAN/subnet, isolated with firewalls?  Did he SPAN every single port to an IDS/IPS?  If not, what's the difference here?  The Hypervisor?  What protection mechanisms has the fancy virtual firewall put in place to protect it?  None.

[Customer] was increasingly concerned about the risks of virtual networks, which range from security policy violations such as mixing trusted and un-trusted systems to malware exploits that can propagate undetected within a virtual network. 

Based upon the second paragraph above where the [Customer] admitted they put their virtualized environment into production without visibility or security, they clearly weren't that concerned with the risks.

A large amount of data center network traffic was moving between VMs and [Customer] had no visibility or control over the communication on the virtual network.

So if there were no security or visibility tools in place, how was it determined that traffic was moving between VM's?

Does this mean that all the customer's VM's were in a single VLAN and not segmented? If not and vSwitch configurations via port groups and VLANs were configured around VM criticality, role or function, then they certainly had some insight into what was moving between VM's and the "data center," right?

I must be confused. 

[Customer's] traditional network security tools could not monitor, analyze or troubleshoot inter-VM traffic because communications between VMs on the same physical host never touch the traditional network. 

Assuming that the VM's weren't in a single VLAN/portgroup on a single vSwitch and instead were segmented via VLANs/subnets, then the only way to get traffic from VLAN/IP Subnet A to VLAN/IP Subnet B (and thus VM A to VM B in these VLAN's) is though a layer 3 routing process which generally means traffic exits the virtual network and hits the physical network...where said "traditional security tools" could see it.

Of course, this doesn't help intra-VM traffic on the same portgroup/VLAN/vSwitch, but that's not what they pointed to above, but assuming they don't look at inter-machine traffic in their physical network on the same VLAN, again I ask what's the difference?

VMs were able to communicate with each other without observation or policy-based inspection and filtering, which left them highly vulnerable to malicious exploits. Additionally, worms and viruses could further spread among physical hosts via unintentional VMotion of an infected VM.

Back to my point above about how the [Customer] monitored traffic between physical hosts...if you don't do it in physical environments, why the fret in the virtual?

Oh and "unintentional VMotion!?" ZOMG!  For a VM to be "infected," excluding direct physical access, wouldn't the threat vector be the network in the first place?  

The [Vendor] virtual firewall was specifically created to mitigate the risks of virtual networks, while maintaining the ROI of virtualization.

What "risks of virtual networks" does this product mitigate in the absence of vulnerability or clearly defined threats that aren't faced in the physical realm?  Let me tell you.  It goes back to the very valid claim that you get better visibility given the integration with the virtualization platform configuration managers to call attention to when CHANGE occurs.

This is the real value of products like this from [Vendor.]  In the long run, the big boys who make mature firewalls and IPS products will get to harness API's like VMsafe and combined with the compartmentalization and segmentation capabilities of vShield Zones leaves a very short runway for products like this.

I'm not suggesting that products like this from [Vendor] don't offer value and solve an immediate pain point. I'd even consider deploying them to solve very specific problems I might have, but then again, I know what problem I'd be trying to solve. ROI?  Oy.

However, unlike the picture painted of the [Customer] above, I plan a little better and understand the impact virtualization has on my security posture and how that factors into my assessment and management of risk BEFORE I put it into production.  You should too.

</rant>

{Ed: I use 'intra-' instead of 'inter-' to reflect the "internal" passing of traffic between VM's using the vSwitch. Should traffic exit the vSwitch/host and hit the network as part of interchange between two VM's, I'd count this as 'inter-" VM traffic.}

Sun vs. Cisco? I'm Getting My Popcorn...

Scott Lowe wrote an interesting blog today wondering if Sun was preparing to take on Cisco in the virtualization space, referencing the development of virtualized networking functionality featuring the novel combination of commodity hardware and open source software to unseat the Jolly Green Giant:

A while back in Virtualization Short Take #25 I briefly mentioned Sun’s Crossbow network virtualization software, which brings new possibilities to the Solaris networking world. Not being a Solaris expert, it was hard for me at the time to really understand why Solaris fans were so excited about it; since then, though, I’ve come to understand that Crossbow brings to Solaris the same kind of full-blown virtual network interfaces and such that I use daily with VMware ESX. Now I’m beginning to understand why people are so thrilled!

In any case, an astute reader picked up on my mention of Crossbow and pointed me to this article by Jonathan Schwartz of Sun, and in particular this phrase:

You’re going to see an accelerating series of announcements over the coming year, from amplifying our open source storage offerings, to building out an equivalent portfolio of products in the networking space…

That seemingly innocuous mention was then coupled with this blog post and the result was this question: is Sun preparing to take on Cisco? Is Sun getting ready to try to use commodity hardware and open source software to penetrate the networking market in the same way that they are using commodity hardware and open source software to try to further penetrate the storage market with their open storage products (in particular, the 7000 series)?

It’s an interesting thought, to say the least. Going up against Cisco is a bold move, though, and I question Sun’s staying power in that sort of battle. Of course, with Cisco potentially distracted by the swirling rumors regarding the networking giant’s entry into the server market, now may be the best time to make this move.

It's really the last paragraph that is of interest to me, specifically the boldfaced sentence I highlighted.  I think the "rumors" have pretty much been substantiated by the mainstream press, so let's assume "California" is going to happen.

Let's make a couple of things really, really clear:

1. I don't know how anyone can think that Cisco is "distracted" by bringing to market the logical extension of virtualized infrastructure -- the compute function -- as anything other than a shrewd business decision to offer a complete end-to-end solution to customers.  I talked about it here in blog post titled "Cisco Is NOT Getting Into the Server Business..." This is an Enterprise Architecture play, pure and simple.
2. Honestly, if we're discussing commoditization, a server is a server is a server, whether it's in a blade form factor or not, and it's not like Cisco has to worry about building things from scratch. The availability of OEM/ODM components (raw or otherwise) means they don't have to start from scratch.  Oh yes, I know HP spent a bazillion dollars on C-Class fan engineering and IBM's BCHT is teh awesome and...
3. The whole game is Unified Computing; bringing together enterprise class compute, network and storage as a solution with integrated virtualization, management and intelligence; you take the biggest pain point out of the equation -- integration -- and you drive down cost while increasing utility, agility and efficiency.
4. If you look at what "California" is slated to deliver it's hard to see how Sun would compete: A blade based chassis with integrated Nexus converged networking/storage, integrated virtualization from VMware (with Nexus/VN-Link,) and management from BMC.  You know, Enterprise stuff, not integration hodge podge. 

So, I ask, does this look like a distraction to you? 

I'm not knocking Sun (or Scott to be clear,) but if I were they, I'd be much more worried about HP or IBM or even Microsoft and Redhat.

I'm grabbing my popcorn, but this battle might be over before the kernels (ha!) start popping.

/Hoff

If Virtualization is a Religion, Does That Make Cloud a Cult?

I had just finished reading Virtual Gipsy's post titled "VMware as religion" when my RSS reader featured a referential post from VM/ETC's Rich titled "vTheology: the study of virtualization as religion."

While I appreciated the humor surrounding the topic, I try never to mix friends politics, and religion* so I'll not wade into the deep end on this one except to suggest what my title asks: 

If virtualization is a religion, does that make cloud a cult?

If so, to whom do I send my tidings?  Who is the Cardinal of the Cloud?  The Pope of PaaS?  The Shaman of Service?

...and where's my phenobarbital, vodka and purple cape?

/Hoff

*...and truth be told, I'm not feeling particularly witty this morning.

I'm Sorry, But Did Someone Redefine "Open" and "Interoperable" and Not Tell Me?

I've got a problem with the escalation of VMware's marketing abuse of the terms "open," "interoperable," and "standards."  I'm a fan of VMware, but this is getting silly.

When a vendor like VMware crafts an architecture, creates a technology platform, defines an API, gets providers to subscribe to offering it as a service and does so with the full knowledge that it REQUIRES their platform to really function, and THEN calls it "open" and "interoperable," because an API exists, it is intellectually dishonest and about as transparent as saran wrap to call that a "standard" to imply it is available regardless of platform.

We are talking about philosophically and diametrically-opposed strategies between virtualization platform players here, not minor deltas along the bumpy roadmap highway.  What's at stake is fundamentally the success or failure of these companies.  Trying to convince the world that VMware, Microsoft, Citrix, etc. are going to huddle for a group hug is, well, insulting.

This recent article in the Register espousing VMware's strategy really highlighted some of these issues as it progressed. Here's the first bit which I agree with:

There is, they fervently say, no other enterprise server and data centre virtualisation play in town. Businesses wanting to virtualise their servers inside a virtualising data centre infrastructure have to dance according to VMware's tune. Microsoft's Hyper-V music isn't ready, they say, and open source virtualisation is lagging and doesn't have enterprise credibility.

Short of the hyperbole, I'd agree with most of that.  We can easily start a religious debate here, but let's not for now.  It gets smelly where the article starts talking about vCloud which, given VMware's protectionist stance based on fair harbor tactics, amounts to nothing more (still) than a vision.  None of the providers will talk about it because they are under NDA.  We don't really know what vCloud means yet: 

Singing the vcloud API standard song is very astute. It reassures all people already on board and climbing on board the VMware bandwagon that VMware is open and not looking to lock them in. Even if Microsoft doesn't join in this standardisation effort with a whole heart, it doesn't matter so long as VMware gets enough critical mass.

How do you describe having to use VMware's platform and API as VMware "...not looking to lock them in?" Of course they are!  

To fully leverage the power of the InterCloud in this model, it really amounts to either an ALL VMware solution or settling for basic connectors for coarse-grained networked capability.

Unless you have feature-parity or true standardization at the hypervisor and management layers, it's really about interconnectivity not interoperability.  Let's be honest about this.

By having external cloud suppliers and internal cloud users believe that cloud federation through VMware's vCloud infrastructure is realistic then the two types of cloud user will bolster and reassure each other. They want it to happen and, if it does, then Hyper-V is locked out unless it plays by the VMware-driven and VMware partner-supported cloud standardisation rules, in which case MIcrosoft's cloud customers are open to competitive attack. It's unlikely to happen.

"Federation" in this context really only applies to lessening/evaporating the difference between public and private clouds, not clouds running on different platforms.  That's, um, "lock-in."

Standards are great, especially when they're yours. Now we're starting to play games.  VMware should basically just kick their competitors in the nuts and say this to us all:

"If you standardize on VMware, you get to leverage the knowledge, skills, and investment you've already made -- regardless of whether you're talking public vs. private.  We will make our platforms, API's and capabilities as available as possible.  If the other vendors want to play, great.  If not, your choice as a customer will determine if that was a good decision for them or not."

Instead of dancing around trying to muscle Microsoft into playing nice (which they won't) or insulting our intelligence by handwaving that you're really interested in free love versus world domination, why don't you just call a spade a virtualized spade.

And by the way, if it weren't for Microsoft, we wouldn't have this virtualization landscape to begin with...not because of the technology contributions to virtualization, but rather because the inefficiencies of single app/OS/hardware affinity using Microsoft OS's DROVE the entire virtualization market in the first place!

Microsoft is no joke.  They will maneuver to outpace VMware. HyperV and Azure will be a significant threat to VMware in the long term, and this old Microsoft joke will come back to haunt to VMware's abuse of the words above:

Q: How many Microsoft engineers does it take to change a lightbulb?  
A: None, they just declare darkness a standard.

is it getting dimmer in here?

/Hoff

Internal v. External/Private v. Public/On-Premise v. Off- Premise: It's all Cloud But How You Get There Is Important.

I've written about the really confusing notional definitions that seem to be hung up on where the computing actually happens when you say "Cloud:" in your datacenter or someone else's.  It's frustrating to see how people mush together "public, private, internal, external, on-premise, off-premise" to all mean the same thing.

They don't, or at least they shouldn't, at least not within the true context of Cloud Computing.

In the long run, despite all the attempts to clarify what we mean by defining "Cloud Computing" more specifically as it relates to compute location, we're going to continue to call it "Cloud."  It's a sad admission I'm trying to come to grips with.  So I'll jump on this bandwagon and take another approach.

Cloud Computing will simply become ubiquitous in it's many forms and we are all going to end up with a hybrid model of Cloud adoption -- a veritable mash-up of Cloud services spanning the entire gamut of offerings.  We already have today.

Here are a few, none-exhaustive examples of what a reasonably-sized enterprise can expect from the move to a hybrid Cloud environment:

1. If you're using one or more SaaS vendors who own the entire stack, you'll be using their publicly-exposed Cloud offerings.  They manage the whole kit-and-kaboodle, information and all. 
2. SaaS and PaaS vendors will provide ways of integrating their offerings (some do today) with your "private" enterprise data stores and directory services for better integration and business intelligence.
3. We'll see the simple evolution of hosting/colocation providers add dynamic scalability and utility billing and really push the Cloud mantra.  
4. IaaS vendors will provide (ala GoGrid) ways of consolidating and reducing infrastructure footprints in your enterprise datacenters by way of securely interconnecting your private enterprise infrastructure with managed infrastructure in their datacenters. This model simply calls for the offloading of the heavy tin. Management options abound: you manage it, they manage it, you both do...
5. Other IaaS players will continue to offer a compelling suite of soup-to-nuts services (ala Amazon) that depending upon your needs and requirements, means you have very little (or no) infrastructure to speak of.  You may or may not be constrained by what you can or need to do as you trade of flexibility for conformity here.
6. Virtualization platform providers will no longer make a distinction in terms of roadmap and product positioning between internal/external or public/private. What is enterprise virtualization today simply becomes "Cloud."  The same services, split along virtualization platform party lines, will become available regardless of location. 
7. This means that vendors who today offer proprietary images and infrastructure will start to drive or be driven to integrate more open standards across their offerings in order to allow for portability, interoperability and inter-Cloud scalability...and to make sure you remain a customer.
8. Even though the Cloud is supposed to abstract infrastructure from your concern as a customer, brand-associated moving parts will count; customers will look for pure-play vetted integration between the big players (networking, virtualization, storage) in order to fluidly move information and applications into and out of Cloud offerings seamlessly 
9. The notion of storage is going to be turned on its head; the commodity of bit buckets isn't what storage means in the Cloud.  All the chewy goodness will start to bubble to the surface as value-adds come to light: DeDup, backup, metadata, search, convergence with networking, security...
10. More client side computing will move to the cloud (remember, it doesn't matter whether it's internal or external) with thin client connectivity while powerful smaller-footprint mobile platforms (smartphones/netbooks) with native virtualization layers will also accelerate in uptake

Ultimately, what powers your Cloud providers WILL matter.  What companies adopt internally as their virtualization, networking, application delivery, security and storage platforms internally as they move to consolidate and then automate will be a likely choice when evaluating top-rung weighting when they identify what powers many of their Cloud providers' infrastructure.

If a customer can take all the technology expertise, the organizational and operational practices they have honed as they virtualize their internal infrastructure (virtualization platform, compute, storage, networking, security) and basically be able to seamlessly apply that as a next step as the move to the Cloud(s), it's a win.

The two biggest elements of a successful cloud: integration and management. Just like always.

I can't wait.

/Hoff

*Yes, we're concerned that if "stuff" is outside of our direct control, we'll not be able to "secure" it, but that isn't exactly a new concept, nor is it specific to Cloud -- it's just the latest horse we're beating because we haven't made much gains in being able to secure the things that matter most in the ways most effective for doing that.

Virtualization & Security: Disruptive Technologies - A Four Part Video Miniseries...

About nine months ago, Dino Dai Zovi, Rich Mogull and I sat down for about an hour as Dennis Fisher from TechTarget interviewed us in a panel style regarding the topic of virtualization and security.  It has just been released now.

Considering it was almost a lifetime ago in Internet time, almost all of the content is still fresh and the prognostication is pretty well dead on.

Enjoy:

Part 1: The Greatest Threats to Virtualized Environments

Part 2: The Security Benefits of Virtualization

Part 3: The Organizational Challenges of Virtualization

Part 4: Virtualization and Security Vendors

/Hoff

P.S. The camera adds like 40 pounds, really ;)

Incomplete Thought: Separating Virtualization From Cloud?

I was referenced in a CSO article recently titled "Four Questions On Google App Security." I wasn't interviewed for the story directly, but Bill Brenner simply referenced our prior interviews and my skepticism for virtualization security and cloud Security as a discussion point.

Google's response was interesting and a little tricky given how they immediately set about driving a wedge between virtualization and Cloud.  I think I understand why, but if the article featured someone like Amazon, I'm not convinced it would go the same way...

As I understand it, Google doesn't really leverage much in the way of virtualization (from the classical compute/hypervisor perspective) for their "cloud" offerings as compared to Amazon. That may be in large part due to the fact of the differences in models and classification -- Amazon AWS is an IaaS play while GoogleApps is a SaaS offering.

You can see why I made the abstraction layer in the cloud taxonomy/ontology model "optional."

This post dovetails nicely with Lori MacVittie's article today titled "Dynamic Infrastructure: The Cloud Within the Cloud" wherein she highlights how the obfuscation of infrastructure isn't always a good thing. Given my role, what's in that cloudy bubble *does* matter.

So here's my incomplete thought -- a question, really:

How many of you assume that virtualization is an integral part of cloud computing? From your perspective do you assume one includes the other?  Should you care?

Yes, it's intentionally vague.  Have at it.

/Hoff

Old MacDonald Had a (Virtual Server) Farm, I/O, I/O, Oh!

It's all about the I/O and your ability to shuffle packets...or see them in the first place...

In reading Neil macDonald's first post under the Gartner-branded blog titled "Virtualization Security Is Transformational -- If the legacy Security Vendors Would Stop Fighting It," I find myself nodding in violent agreement whilst also shaking my head in bewilderment.  Perhaps I missed the point, but I'm really confused.

Neil sets the stage by suggesting that "established" security vendors who offer solutions for non-virtualized environments simply "...don't get it" when it comes to realizing the shortcomings of their existing solutions in virtualized contexts and that they are "fighting" the encroachment of virtualization on their appliance sales:

Many are clinging to business models based on their overpriced hardware-based solutions and not offering virtualized versions of their solutions. They are afraid of the inevitable disruption (and potential cannibalization) that virtualization will create. However, you and I have real virtualization security needs today and smaller innovative startups have rushed in to fill the gap. And, yes, there are pricing discontinuities. A firewall appliance that costs $25,000 in a physical form can cost $2500 or less in a virtual form from startups like Altor Networks or Reflex Systems.

I'm very interested in which "established" vendors are supposedly clinging to their overpriced hardware-based solutions and avoiding virtualization besides niche players in niche markets that are hardware-bound.  

As far as I can tell the top five vendors by revenue in the security space (that sell hardware, not just software) are all actively engaged in both supporting these environments with the limitations that currently exist based on the virtualization platforms today and are very much investing in development of new solutions to work properly in virtual environments given the unique requirements thereof.

Neil is really comparing apples to muffler brackets.  He points out in his blog that physical appliances can offer multi-gigabit performance whereas software-based VA's cannot, and yet we're surprised that pricing differentials in orders of magnitude exist?  You get what you pay for.

As I pointed out in my Four Horsemen presentation (and is alluded to in the remainder of Neil's post below) EVERY SINGLE VENDOR is currently hamstrung by the same level of integration and architectural limitations involved with the current state of virtual appliance performance in the security space, including those he mentions such as Altor and Reflex.  They are all in a holding pattern.  I've written about that numerous times.

In fact, as I mentioned in my post titled "Visualization Through Virtualization", the majority of these new-fangled, virtualization-specific "security" tools are actually (now) more focused on visibility, management and change montoring/control than they are pure network-level security because they cannot compete from a performance and scalability perspective with hardware-based solutions.

Here's where I do agree with Neil, based upon what I mention above: 

Feature-wise, the security protection services delivered are similar. But, there is a key difference — throughput. What the legacy security vendors forget is that there is still a role for dedicated hardware. There is no way you are going to get full multi-gigabit line speed deep-packet inspection and protocol decode for intrusion prevention from a virtual appliance. A next-generation data center will need both physical and virtualized security controls — ideally, from a vendor that can provide both. I’ll argue that the move to virtualize security controls will grow the overall use of security controls. 

So this actually explains the disparity in both approach and pricing that he alluded to above.  How does this represent vendors "fighting" virtualization?  I see it as hanging on for as long as possible to preserve and milk their investment in the physical appliances Neil says we'll still need while they perform the R&D on their virtualized versions.  They can't deploy the new solutions until the platform to support them exists!

The move to virtualize security controls reduces barriers to adoption. Rather than a sprinkle a few physical appliance here and there based on network topology, we can now place controls when and where they are needed, including physical appliances as appropriate. If fact, the legacy vendors have a distinct advantage over virtualization security startups since you prefer a security solution that spans both your physical and virtual environments with consistent management.

Exactly.  So again, how is this "fighting" virtualization?  

Here's where we ignore reality again:

Over the past six months, I’ve seen signs of life from the legacy physical security vendors. However, some of the legacy physical security vendors have simply taken the code from their physical appliance and moved it into a virtual machine. This is like wrapping a green-screen terminal application with a web front end — it looks better, but the guts haven’t changed. In a data center where workloads move dynamically between physical servers and between data centers, it makes no sense to link security policy to static attributes such as TCP/IP addresses, MAC addresses or servers. 

First of all, what we're really talking about in the enterprise space is VMware, since given its market dominance, this is where the sweet spot is for security vendors.  This will change over time, but for now, it's VMware.

That being the case, the moment VMsafe was announced/hinted at two years ago, 20+ security vendors -- big and small -- have been diligently working within the constructs of what is made available from VMware to re-engineer their products to take advantage of the API's that will be coming in VMware's upcoming release.  This is no small feat.  Distributed virtual switching and the two-tier driver architecture with DVfilters means re-engineering your products and approach.

Until VMware's next platform is released, every security vendor -- big or small -- is hamstrung by having to do exactly what Neil says; creating a software instantiation of their hardware products which is integration-limited for the reasons I've already stated.  What should vendors do?  Firesale their inventories and wait it out?  

I ask again: how is this "fighting" virtualization?

The reason there hasn't been a lot of movement is because the entire industry is in a holding pattern. Pretending otherwise is absolutely ridiculous.  The obvious exception is Cisco which has invested in and developed substantial solutions such as the Nexus 1000v and VN-Link (which is again awaiting the availability of VMware's next release.)

Security policy in a virtualized environment must be tied to logical identities - like identities of VM workloads, identities of application flows and identities of users. When VMs move, policies need to move. This requires more than a mere port of an existing solution, it requires a new mindset.

Yep.  And most of them are adapting their products as best they can.  Many companies will follow the natural path of consolidation and wait to buy a startup in this space and integrate it...much like VMware did with BlueLane, for example.  Others will look to underlying enablers such as Cisco's VN-Link/Nexus 1000v and chose to integrate at the virtual networking layer there and/or in coordination with VMsafe.

The legacy vendors need to wake up. If they don’t offer robust virtualization security capabilities (and, yes, potentially cannibalize the sales of some of their hardware), another vendor will. With virtualization projects on the top of the list of IT initiatives for 2009, we can’t continue to limp along without protection. It’s time to vote with our wallets and make support of virtual environments a mandatory part of our security product evaluation and selection.

Absolutely!  And every vendor -- big and small -- that I've spoken to is absolutely keen on this concept and are actively engaged in developing solutions for these environments with these unique requirements in mind. Keep in mind that VMsafe is about more than just network visibility via the VMM, it also includes disk, memory and CPU...most network-based appliances have never had this sort of access before (since they are NETWORK appliances) and so OF COURSE products will have to be re-tooled.

Overall, I'm very confused by Neil's post as it seems quite contradictory and at odds with what I've personally been briefed on by vendors in the space and overlooks the huge left turns being made by vendors over the last 18 months who have been patiently waiting for VMsafe and other introspection capabilities of the underlying platforms.

I think the windshield needs cleaning on the combine harvester...

/Hoff

PCI Security Standards Council to Form Virtualization SIG...

I'm happy to say that there appears to be some good news on the PCI DSS front with the promise of a SIG being formed this year for virtualization.  This is a good thing. 

You'll remember my calls for better guidance for both virtualization and ultimately cloud computing from the council given the proliferation of these technologies and the impact they will have on both security and compliance.

In that light, news comes from Troy Leach, technical director of the PCI Security Standards Council via a kind note to me from Michael Hoesing:

A PCI SSC Special Interest Group (SIG) for virtualization is most likely coming this year but we don't have any firm dates or objectives as of yet.  We will be soliciting feedback from our Participating Organizations which is comprised of more than 500 companies (which include Vmware, Microsoft, Dell, etc) as well as industry subject matter experts such as the 1,800+ security assessors that currently perform assessments as either a Qualified Security Assessor or Approved Scanning Vendor (ASV).

The PCI SSC Participating Organization program allows industry stakeholders an opportunity to provide feedback on all standards and supporting procedures.  Information to join as a Participating Organization can be found here on our website.

This is a good first step.  if you've got input, make sure to contribute!

/Hoff

Hoff's Upcoming VirtSec/CloudSec Presentations in 2009

I'll be updating my speaking itinerary shortly, but I wanted to let folks know I'm working on three major VirtSec/CloudSec presentations for 2009:
 
The Frogs Who Desired a King

The Frogs Who Desired a King is based on the topical reference to one of Aesop's fable about a discontented population of frogs who appealed to Zeus for a king.

Ultimately, through a comedy of errors, the frogs finally got their new king -- a stork -- which promptly ate them.

We, as a discontented legion of frogs, decry our dark overlords' choices (or lack thereof) of security in virtualized and cloud computing environments and long for security solutions to magically solve all our problems. 

Just like the frogs, we better be careful what we wish for, as our prayers might just be answered, consuming us all. This is the sequel to my "Four Horsemen of the Virtualization Security Apocalypse" series.

---

Cloudifornication: Indiscriminate Information Intercourse Involving Internet Infrastructure

What was in is now out. 

This metaphor holds true not only as an accurate analysis of adoption trends of disruptive technology and innovation, but also parallels the amazing velocity of how our datacenters are being re-perimiterized and quite literally turned inside out.

One of the really scary things happening with the massive convergence of cloud computing is its effect on security models and the information they seek to protect.

Where and how our data is created, processed, accessed, stored, backed up in what is sure to become massively overlaid cloud-based services -- and by whom and whose infrastructure -- yields significant concerns related to security, privacy, compliance and survivability. 

This "infrastructure intercourse" makes it very interesting to try and secure your assets when you don't own the infrastructure and in many cases cannot provide the same levels of functionality we can today.

---

Mozart's "The Marriage of Figaro": Complexity & Insecurity Of the Cloud

Mozart's sequel to the Barber of Seville was lauded as one of the most profound works of its time. 

Its staggering complexity, inviting overtures, rich textures and variety of orchestration were perceived by many as unapproachable, unfathomable and in some cases unintelligible. 

Yet so remarkable and unique was the composition that people flocked to its performances although in many cases were blinded by the simplicity of its underlying complexity.

Such are the parallels with the deeply profound cacophony surrounding the issues of securing Cloud Computing and the tonal miscues hidden amongst its various acts.

This presentation will review the most pressing security, privacy, sustainability and resiliency
issues surrounding the marriage of convenience, economics and computing.

The Quandary Of the Cloud: Centralized Compute But Distributed Data

Here's a theme I've been banging around for quite some time as it relates to virtualization, cloud computing and security.  I've never really sat down and written about it, however.

As we trend towards consolidating and (re)centralizing our computing platforms -- both endpoints and servers -- using virtualization and cloud computing as enablers to do so, we're also simultaneously dealing with the decentralization and distributed data sets that come with technologies such as Web2.0, mobility and exposure of APIs from cloud platforms.*

So here we are all frothed up as virtualization and cloud computing have, in a sense, led us back to the resource-based consolidation of the mainframe model with all it's centralized splendor and client virtualization/thin clients/compartmentalized remote access is doing the same thing for endpoints. 

But the interesting thing is that with Moore's Law, the endpoints are also getting more and more powerful even though we're dumbing them down and trying to make their exposure more limited despite the fact that they can still efficiently process and store data locally.

These models, one could argue, are diametrically opposed when describing how to secure the platforms versus the information that resides on or is utilized by them.  As the cyclic waffling between centralized versus distributed continues, the timing of how and where we adapt to securing them always lags behind.  Which do we focus on securing and where?  The host, centralized server, network.

The unfortunate answer is always "yes."

Remember this (simplified) model of how/where we secure things?

If you juxtapose the image above mentally with how I represent the centralized <--> distributed trends in IT below, it's no wonder we're always behind the curve.  The computing model technology changes much more quickly than the security technology and processes do, thus the disconnect:

I need to update the diagram above to split out the "computing" layer into client and server as well as extend the data layer to reference storage modalities also, but it gets the job done.

At any rate, it's probably obvious and common sense, but when explaining to people why I spend my time pointing out gaps with security in virtualization and cloud models, I found this useful.

/Hoff

* It's important to note that while I refer to/group cloud computing models as centralized, I understand they have a distributed element to them, also.  I would ask you to think about the multiple cloud overlays as centralized resources, regardless of how intrinsically "distributed" in processing/load balancing they may be.

P.S. I just saw an awesome post titled "The Rise of the Stupid Endpoint" on the vinternals blog that shares many of the same points, although much more eloquently.  Check it out here.  Awesome!

Virtualization: An Excuse for Shitty Operating System Software Support

In honor of my friend @quine on Twitter who today complained thusly:

In case you're reading this with Lynx (you web pimp, you!,) Zach was lamenting the fact that vendors who don't support customer operating systems of choice are simply sloughing off development efforts and support by suggesting that customers should simply run it as a VM instead.

Ah, it used to be called "software," but now it's a "virtual appliance!"  Silly rabbit, tricks are for kids.

One might suggest this is a perfectly reasonable use of virtualization technology -- neigh one of the very purposes behind its genesis.  I'd agree, to a point.  However, I've noticed an alarming uptake recently by product managers who are simply short-cutting roadmap/development paths by taking the "lazy" way out.

Hey, it cuts down support, testing, regression and troubleshooting...for the vendor.  But in my favorite commentary, it's simply a "squeezing the balloon problem" because it surfaces a whole host of other issues such as performance, scale, and in some cases support for various virtualization platforms.

What say you?  Do you see this happening more in your enterprise?  Do you care?  Is it a good thing?

/Hoff

Virtualization? So last Tuesday.

This post contains nothing particularly insightful other than a pronounced giant sucking sound that's left a vacuum in terms of forward motion regarding security and virtualization.

Why?

Three things:

1. There's an awful lot of focus moving from the (cough) mature space of server virtualization to the myriad of options and solutions on client virtualization as we're seeing the transition of where we focus our efforts swing again.  

   We're in the throes of yet another "great awakening" where we some of us realize that (gasp!) it's the information we ought to secure and that the platforms themselves are insecure and should be treated as such.  However, we've got so much security invested in the network and servers that we play ping-pong between securing them, bypassing the crown jewels.

   Virtualization has just reinforced that behavior and as we take stock of where we are in (not) securing these vectors looking for the next silver bullet, we knee jerk back to the the conduit through which the user interacts with our precious data: the client.

   The client, it seems, is the focus yet again, driven mostly by economics.  It's interesting to note that even though the theme of RSA this last go-round was "Information Centricity"  someone didn't get the memo. 

   Check out this graphic from my post a ways back titled "Security Will Not End Up In the Network..." for why this behavior is not only normal but will unfortunately lead us to always focus on the grass which turns out not to be greener on the other side.  I suppose I should really break out the "host" into server and client, accordingly:



Further, and rightfully so, the accelerated convergence of storage and networking thanks to virtualization is causing heads to a-splode in ways that cause security to be nothing more than a shrug and a prayer.  What it means to "secure the cloud" is akin to pissing in the wind at the moment.  Hey, if you've got to go, you've got to go...

2. ISV's are in what a amounts to a holding platform waiting for VDCOS, VI4, vSphere with vNetworking and the VMsafe API's to be released so they can unleash their next round of security software appliances to tackle the problems highlighted in my Four Horsemen of the Virtualization Security Apocalypse series.  For platforms other than VMware, we've seen bupkis as it relates to innovation of VirtSec.  
3. The "Cloud" has assimilated us all and combined with the stalling function above, has left us waffling in ambivalence.  The industry is so caught up in the momentum of this new promised revenue land that the blinding opportunity combined with a lack of standards and a slew of new business and technology models means that innovation is being driven primarily by startups while existing brands jockey to retool.

It's messy.  It's going to get messier, but the good news is that it's a really exciting time.  We're going to see old friends like IAM, IDP, VPNs, and good old fashioned routing and switching tart themselves up, hike up the hemlines and start trolling for dates again as virtualization 2.x, VirtSec and Cloud/Cloud Security make all the problems we haven't solved (but know we need to) relevant and pressing once again.

All those SysAdmin and NetAdmin skills you started with before you became a "security professional" will really help in sorting through all this mud.

There exist lots of opportunity to make both evolutionary and revolutionary advancements in solving many of the problems we've been suffering from for decades.  Let's work to press forward and not lose sight of where we're going and more importantly from whence we've come.

/Hoff

  

Servers and Switches and VMM's, Oh My! Cisco's California "Server Switch"

From the desk of Cisco's "Virtualization Wow!" Campaign: When is a switch with a server not a virtualization platform?  When it's a server with a switch as a virtualization platform, of course! ;)

I can't help but smile at the announcement that Cisco is bringing to market a blade-based chassis which bundles together Intel's Nehalem-based server processors, the Nexus 5000 switch, and VMware's virtualization and management platform.  From InformationWeek:

Cisco's system, code-named California, likely will be introduced in the spring, according to the sources. It will meld Cisco's Nexus 5000 switch that converges storage and data network traffic, blade servers that employ Intel Nehalem processors, and virtualization management with help from VMware. 

This news was actually broken back in the beginning of December by virtualization.info but I shamefully missed it.  It looked like a bunch of others did, too.

This totally makes sense as virtualization has driven convergence across the compute, network and storage realms and has highlighted the fact that the provisioning, automation, and governance -- up and down the stack -- demands a unified approach for management and support.

For me, this is the natural come-about of what I wrote about in July of 2007 in a blog post titled "Cisco & VMware - The Revolution Will Be...Virtualized?":

This [convergence of network, compute and virtualization, Ed.] is interesting for sure and if you look at the way in which the demand for flexibility of software combined with generally-available COTS compute stacks and specific network processing where required, the notion that Cisco might partner with VMWare or a similar vendor such as SWSoft looks compelling.  Of course with functionality like KVM in the Linux kernel, there's no reason they have to buy or ally...

Certainly there are already elements of virtualization within Cisco's routing, switching and security infrastructure, but many might argue that it requires a refresh in order to meet the requirements of their customers.  It seems that their CEO does.

When I last blogged about Cisco's partnership with VMware and (what is now called) the Nexus 1000v/VN-Link, I made reference to the fact that I foresaw the extraction of the VM's from the servers and suggested that we would see VM's running in the Nexus switches themselves.  Cisco representatives ultimately put a stake in the sand and said this would never happen in the comments of that blog post.

Now we know what they meant and it makes even more sense.

So the bundling of the Nexus 5000* (with the initiator,) the upcoming protocol for VM-Flow affinity tagging, the integrated/converged compute and storage capabilities, and Intel's SR-IOV/MR-IOV/IOMMU technologies in the Nehalem, all supported by the advances with vNetworking/VN-Link makes this solution a force to be reckoned with.

Other vendors, especially those rooted in servers and networking such as HP and IBM, are likely to introduce their own solutions, but given the tight coupling of the partnership, investment and technology development between Intel, VMware and Cisco, this combo will be hard to beat. 

Folks will likely suggest that Cisco has no core competency in building, selling or supporting "servers," but given the channel and partnership with VMware -- with virtualization abstracting that hardware-centric view in the first place -- I'm not sure this really matters.  We'll have to see how accurate I am on this call.

Regardeless of the semantic differences of where the CPU execution lives (based on my initial prediction,) all the things I've been talking about that seemed tangentially-related but destined to come together seem to have.  Further, here we see the resurgence (or at least redefinition of Big Iron, all over again...)

Remember the Four Horsemen slides and the blog post (the network is the computer, is the network, is the...) where I dared you to figure out where "the network" wasn't in the stack?  This is an even more relevant question today. 

It's going to be a very interesting operational and organizational challenge from a security perspective when your server, storage, networking and virtualization platform all come from a single source.

California Dreamin'...

/Hoff

* Not that the 1000v ain't cool, but that little slide that only appeared once at VMworld about the 5000v and the initiator was just too subtely delicious not to be the real juice in the squeeze. The 1000v obviously has its place and will be a fantastic solution, but for folks who are looking for a one-stop shop for their datacenter blueprint designs heavily leveraging virtualization, this makes nothing but sense.

Rogue VM Sprawl? Really?

I keep hearing about the impending doom of (specifically) rogue VM sprawl -- our infrastructure overrun with the unchecked proliferation of virtual machines running amok across our enterprises.  Oh the horror!

Most of the examples use the consolidation of server VM's onto hosts as delivered by virtualization as their example.

I have to ask you though, given what it takes to spin up a VM on a platform such as VMware, how can you have a "rogue" VM sprawling its way across your enterprise!? 

Someone -- an authorized administrator -- had to have loaded it into inventory, configured its placement on a virtual switch, and spun it up via VirtualCenter or some facsimile thereof depending upon platform.

That's the definition of a rogue?  I can see where this may be a definitional issue, but the marketeers are getting frothed up over this very issue, whispering in your ear constantly about the impending demise of your infrastructure...and undetectable hypervisor rootkits, too.  :)

It may be that the ease of which a VM *can* be spun up legitimately can lead to the overly-exhuberant deployment of VM's without understanding the impact this might have on the infrastructure, but can we please stop grouping stupidity and poor capacity planning/impact analysis with rogueness?  They're two different things.

If administrators are firing off VMs that are unauthorized, unhardened, and unaccounted for, you have bigger problems than that of virtualization and you ought to consider firing them off. 

The inventory of active VMs is a reasonably easy thing to keep track of; if it's running, I can see it. 

I know "where" it is and I can turn it off.  To me, the bigger problem is represented by the offline VMs which can live outside that inventory window, just waiting to be reactivated from their hypervisorial hibernation.

But does that represent "rogue?"

You want an example of a threat which represents truly rogue VM "sprawl" that people ought to be afraid of?  OK, here's one, and it happened to me.  I talk about it all the time and people usually say "Oh, man, I never thought of that..."  usually because we're focused on server virtualization and not the client side.

The Setup: It's 9:10am about 4-5 years ago.  Settling in to read email after getting to work, the klaxon alarms start blaring.  The IDS/IPS consoles start going mad. Users can't access network resources.  DHCP addresses are being handed out from somewhere internally on the network from pools allocated to Korean address space.

We take distributed sniffer traces.  Trackback through firewall, IDS and IPS logs and isolate the MAC address in the CAM tables of the 96 port switch to which the offending DHCP server appears to be plugged, although we can't ping it.

My analyst is now on a mission to unplug the port, so he undocks his laptop and the alarms silence.

I look over at him.  He has a strange look on his face.  He docks his laptop again.  Seconds later the alarms go off again.


The Culprit: Turns out said analyst was doing research at home on our W2K AD/DHCP server hardening scripts.  He took our standard W2K server image, loaded it as a VM in VMware Workstation and used it at home to validate funtionality.

The image he used had AD/DHCP services enabled.

When he was done at home the night before, he minimized VMware and closed his laptop.

When he came in to work the next morning, he simply docked and went about reading email, forgetting the VMW instance was still running.  Doing what it does, it started responding to DHCP requests on the network.

Because he was using shared IP addresses for his VM and was "behind" the personal firewall on his machine which prohibits ICMP requests based on the policy (but obviously not bootp/DHCP) we couldn't ping it or profile the workstation...

Now, that's a rogue VM.  An accidental rogue VM.  Imagine if it were an attacker.  Perhaps he/she was a legitimate user but disgruntled.  Perhaps he/she decided to use wireless instead of wired.  How much fun would that be?

Stop with the "rogue (server) VM" FUD, wouldya?

Kthxbye.

/Hoff

Virtual Routing - The Anti-Matter of Network SECURITY...

Here's a nod to Rich Miller who pointed over (route the node, not the packet) to a blog entry from Andreas Antonopoulos titled "Virtual Routing - The anti-matter of network routing."

The premise, as brought up by Doug Gourlay from Cisco at the C-Scape conference was seemingly innocuous but quite cool:

"How about using netflow information to re-balance servers in a data center"

Routing: Controlling the flow of network traffic to an optimal path between two nodes

Virtual-Routing or Anti-Routing: VMotioning nodes (servers) to optimize the flow of traffic on the network.

Using netflow information, identify those nodes (virtual servers) that have the highest traffic "affinity" from a volume perspective (or some other desired metric, like desired latency etc) and move (VMotion, XenMotion) the nodes around to re-balance the network. For example, bring the virtual servers exchanging the most traffic to hosts on the same switch or even to the same host to minimize traffic crossing multiple switches. Create a whole-data-center mapping of traffic flows, solve for least switch hops per flow and re-map all the servers in the data center to optimize network traffic.

My first reaction was, yup, that makes a lot of sense from a network point of view, and given who made the comment, it does make sense. Then I choked on my own tongue as the security weenie in me started in on the throttling process, reminding me that while this is fantastic from an autonomics perspective, it's missing some serious input variables.

Latency of the "network" and VM spin-up aside, the dirty little secret is that what's being described here is a realistic and necessary component of real time (or adaptive) infrastructure.  We need to get ultimately to the point where within context, we have the ability to do this, but I want to remind folks that availability is only one leg of the stool.  We've got the other nasty bits to concern ourselves with, too.

Let's look at this from two perspectives: the network plumber and the security wonk

From the network plumbers' purview, this sounds like an awesome idea; do what is difficult in non-virtualized environments and dynamically adjust and reallocate the "location" of an asset (and thus flows to/from it) in the network based upon traffic patterns and arbitrary metrics.  Basically, optimize the network for the lowest latency and best performance or availability by moving VM's around and re-allocating them across the virtual switch fabric (nee DVS) rather than adjusting how the traffic gets to the static nodes.

It's a role reversal: the nodes become dynamic and the network becomes more static and compartmentalized.  Funny, huh?

--

The security wonk is unavailable for comment.  He's just suffered a coronary event.  Segmented network architecture based upon business policy, security, compliance and risk tolerances make it very difficult to perform this level of automation via service governors today, especially in segmented network architecture based upon asset criticality, role or function as expressed as a function of (gulp) compliance, let's say. 

Again, the concept works great in a flat network where asset grouping is, for the most part, irrelevant (hopefully governed by a policy asserting such) where what you're talking about is balancing the compute with network and storage, but the moment you introduce security, compliance and risk management as factors into the decision fabric, things get very, very difficult.

Now, if you're Cisco and VMware, the models for how the security engines that apply policy consistently across these fluid virtualized networks is starting to take shape, but what we're missing are the set of compacts or contracts that consistently define and enforce these policies no matter where they move (and control *if* they can move) and how they factor these requirements into the governance layer.

The standardization of governance approaches -- even at the network layer -- is lacking.  There are lots of discrete tools available but the level of integration and the input streams and output telemetry are not complete.

If you take a look, as an example, at CIRBA's exceptional transformational analytics and capacity management solution, replete with their multi-dimensional array of business process, technical infrastructure and resource mapping, they have no input for risk assessment data, compliance or "security" as variables.

When you look at the utility brought forward by the dynamic, agile and flexible capabilities of virtualized infrastructure, it's hard not to extrapolate all the fantastic things we could do. 

Unfortunately, the crushing weight of what happens when we introduce security, compliance and risk management to the dance means we have a more sobering discussion about those realities.

Here's an example reduced to the ridiculous: we have an interesting time architecting networks to maximize throughput, reduce latency and maximize resilience in the face of what can happen with convergence issues and flapping when we have a "routing" problem.

Can you imagine what might happen when you start bouncing VM's around the network in response to maximizing efficiency while simultaneously making unavailable the very resources we seek to maximize the availability of based upon disassociated security policy violations?  Fun, eh?

While we're witnessing a phase shift in how we design and model our networks to support more dynamic resources and more templated networks, we can't continue to mention the benefits and simply assume we'll catch up on the magical policy side later.

So for me, Virtual Routing is the anti-matter of network SECURITY, not network routing...or maybe more succinctly, perhaps security doesn't matter at all?

/Hoff

GigaOm's Alistair Croll on Cloud Security: The Sky Is Falling!...and So Is My Tolerance For Absurdity

I just read the latest blog of Alistair Croll from GigaOm titled "Cloud Security: The Sky Is Falling!" in which he suggests that we pillow-hugging security wonks ought to loosen our death grips on our data because not only are we flapping our worry feathers for nothing, but security in "the Cloud" will result in better security than we have today. 

It's an interesting assertion, really, that despite no innovative changes in the underpinnings of security technology, no advances in security architecture or models and no fundamental security operational enhancements besides the notion of enhanced "monitoring," that simply outsourcing infrastructure to a third party "in the cloud" will in some way make security "better," whatever version of "the Cloud" you may be describing:

I don’t believe that clouds themselves will cause the security breaches and data theft they anticipate; in many ways, clouds will result in better security. Here’s why:

• Fewer humans – Most computer breaches are the result of human error; only 20-40 percent stem from technical malfunctions. Cloud operators that want to be profitable take humans out of the loop whenever possible.
  
• Better tools – Clouds can afford high-end data protection and security monitoring tools, as well as the experts to run them. I trust Amazon’s operational skills far more than my own.
  
• Enforced processes – You could probably get a co-worker to change your company’s IT infrastructure. But try doing it with a cloud provider without the proper authorization: You simply won’t be able to.
  
• Not your employees — Most security breaches are committed by internal employees. Cloud operators don’t work for you. When it comes to corporate espionage, employees are a much more likely target.

Of course it takes people to muck things up, it always has and always will.  Rushing to embrace a "new" computing model without the introduction of appropriately compensating controls, adapted risk assessment/management methodologies and practices will absolutely introduce new threats, vulnerabilities and risk at a pace driven by supposed economic incentives that have people initially foaming at their good fortune and then fuming when it all goes bad.

This comes down to the old maxim: "guns don't kill people, people kill people."  Certainly "the Cloud" alone won't increase breaches and data theft, but using it without appropriate safeguards will.

This is an issue of squeezing the balloon.  The problem doesn't change in volume, it just changes shape.

Those of us concerned about security and privacy in cloud computing models have good reason to be concerned; we live with and have lived with these sorts of disruptive innovations and technology before and it really, really screws things up because the security models and technology we can lean on to manage risk is not adapted to this at all and the velocity of change eclipses our ability to do do our jobs competently.

Further bonking things up is the very definition of "the Cloud(s)" in the first place.

Despite the obvious differences in business models, use cases, technical architecture as well as the non-existence of a singularity called "The Cloud," this article generalizes and marginalizes the security challenges of cloud computing regardless.  In fact, it emphasizes on one leg of the IT stool (people) to the point of downplaying via the suspension of disbelief that the other two (process and technology) are problems less deserving of attention as they are magically addressed.

To be fair, I can certainly see Alistair's argument holding water within the context of an SME/SMB with no dedicated expertise in security and little or no existing cost burden in IT infrastructure.  The premise: let your outsourced vendor provide you with the expertise in security you don't have as they have a vested interest to do so and can do it better than you.  

The argument hinges on two things: that insiders intent on malicious activity by tampering with "infrastructure" are your biggest risk eliminated by "the cloud" and that infrastructure and business automation, heretofore highly sought after elements of enterprise modernization efforts, is readily available now and floating about in the cloud despite its general lack of availability in the enterprise.

So here's what's amusing to me:

1. It takes humans to operate the cloud infrastructure.  These human operators, despite automation, still suffer from the same scale and knowledge limitations as those in the real world.  Further the service governance layers that translate business process, context and risk into enforceable policy across a heterogeneous infrastructure aren't exactly mature. 
     
2. The notion that better tools exist in the cloud that haven't as yet been deployed in the larger enterprise seems a little unbelievable.  Again, I agree that this may be the case in the SME/SMB, but it's simply not the case in larger enterprises.  Given issues such as virtualization (which not all cloud providers depend upon, but bear with me) which can actually limit visibility and reach, I'd like to understand what these tools are why we havent' heard of them before.


3. The notion that you can get a co-worker to "...change your company's IT infrastructure" but you can't get this same event impact to occur in the cloud is ludicrous.  Besides the fact that the bulk of breaches result from abuse or escalation of privilege in operating systems and applications, not general "infrastructure," and   "the Cloud," having abstracted this general infratructure from view. leaves bare the ability to abuse the application layer just as ripely.


4. Finally, Alaistair's premise that the bulk of attacks originate internally is misleading. Alistair's article was written a few days ago.  The Intranet Journal article he cites to bolster his first point substantiating his claim was written in 2006 and is based upon a study done by CompTIA in 2005.  2005!  That's a lifetime by today's standards. Has he read the Verizon breach study that empirically refutes many of his points? (*See Below in extended post)
   

 As someone who has been on both the receiving end as well as designed and operated managed (nee Cloud) security as a service for customers globally, there are a number of exceptions to Alistair's assertions regarding the operational security prowess in "the Cloud" with this being the most important: 

As "the Cloud" provider adds customers, the capability to secure the infrastructure and the data transiting it, ultimately becomes an issue of scale, too. The more automation that is added, the more false positives show up, especially in light of the fact that the service provider has little or no context of the information, business processes or business impact that their monitoring tools observe.  You can get rid of the low-hanging fruit, but when it comes down to impacting the business, some human gets involved.

The automation that Alastair asserts is one of the most important reasons why Cloud security will be better than non-Cloud security ultimately suffers from the same  lack of eyeballs problem that the enterprise supposedly has in the first place.

For all the supposed security experts huddled around glowing monitors in CloudSOC's that are vigilantly watching over "your" applications and data in the Cloud, the dirty little secret is that they rely on basically the same operational and technical capabilities as enterprises deploy today, but without context for what it is they are supposedly protecting.  Some rely on less.  In fact, in some cases, unless they're protecting their own infrastructure, they don't do it at all -- it's still *your* job to secure the stacks, they just deal with the "pipes."

We're not all Chicken Little's, Alistair.  Some of us recognize the train when it's heading toward us at full speed and prefer not to be flattened by it, is all.

/Hoff

Oh Great Security Spirit In the Cloud: Have You Seen My WAF, IPS, IDS, Firewall...

I'm working on the sequel to my Four Horsemen of the Virtualization Security Apocalypse presentation.

It's called "The Frogs Who Desperately Wanted a King: An Information Security Fable of Virtualization, RTI and Cloud Computing Security." (Okay, it also has the words "interpretive dance" in it, but that's for another time...)

Many of the interesting issues from the Four Horsemen regarding the deficiencies of security solutions and models in virtualized environments carries over directly to operationalizing security in the Cloud. 

As a caveat, let's focus on a cost-burdened "large" enterprise who's involved in moving from physical to virtual to cloud-based services.

I'm not trying to make a habit of picking on Amazon AWS, but it's just such a fantastic example for my point, which is quite simply:

While the cloud allows you to obviate the need for physical compute, network and storage infrastructure, it requires a concerted build-out and potential reinvestment in a software-based security infrastructure which, for most large enterprises, does not consist of the same solutions deployed today.

Why?  Let me paint the picture...

In non-virtualized environments, we generally use dedicated appliances or integrated solutions that provide one of more discrete security functions. 

These solutions are built generally on hardened OS's, sometimes using custom hardware and sometimes COTS boxes which are tarted up.  They are plumbed in between discretely segregated (physical or logical) zones to provide security boundaries defined by arbitrary policies based upon asset classification, role, function, access, etc.  We've been doing this for decades.  It's the devil we know.

In virtualized environments, we currently experience some real pain when it comes to replicating the same levels of security, performance, resiliency, and scale using software-based virtual appliances to take the place of the hardware versions in our physical networks when we virtualize the interconnects within these zones.

There are lots of reasons for this, but the important part is realizing that many of the same hardware solutions are simply not available as virtual appliances and even when they are, they are often not 1:1 in terms of functionality or capability.  Again, I've covered this extensively in the Four Horsemen.

So if we abstract this to its cloudy logical conclusion, and use AWS as the "platform" example, we start to face a real problem for an enterprise that has a decade(s) of security solutions, processes and talent that is focused on globalizing, standardizing and optimizing their existing security infrastructure and are now being forced to re-evaluate not only the technology selection but the overall architecture and operational model to support it.

Now, it's really important that you know, dear reader, that I accept that one can definitely deploy security controls instantiated as both network and host-based instances in AWS.  There are loads of options, including the "firewall" provided by AWS.

However, the problem is that in the case of building an AMI for AWS supporting a particular function (firewall, WAF, IPS, IDS, etc.) you may not have the same solutions available to you given the lack of support for a particular distro, lack of "port" to a VA/VM, or issues surrounding custome kernels, communication protocols, hardware, etc...  You may be limited in many cases to having to rely on open source solutions.

In fact, when one looks at most of the examples given when describing securing AWS instances, they almost always reference OSS solutions such as Snort, OSSEC, etc.  There's absolutely NOTHING wrong with that, but it's only one dimension.

That's going to have a profound effect across many dimensions.  In many cases, enterprises have standardized on a particular solution not because it's special from a "security" perspective, but because it's the easiest to manage when you have lots of them and they are supportable 24/7 by vendors with global reach and SLA's that represent the criticality of their function.

That is NOT to say that OSS solutions are not easy to manage or supportable in this fashion, but I believe it's a valid representation of the state of things.

(Why am I getting so defensive about OSS? ;)

Taking it further, and using my favorite PCI in the Cloud argument, what if the web application firewall that you've spent hundreds of thousands of dollars purchasing, tuning and deploying in support of PCI DSS throughout the corporate enterprise simply isn't available as a software module installable in an AMI in the cloud?  Or the firewall?  Or the IPS? 

In the short term this is a real problem for customers.  In the long term, it's another potential life preserver for security ISV's and an opportunity for emerging startups to think about new ways of solving this problem.

/Hoff

Infrastructure 2.0 and Virtualized/Cloud Networking: Wait, Where's My DNS/DHCP Server Again?

I read James Urquhart's first blog post written under the Cisco banner today titled "The network: the final frontier for cloud computing" in which he describes the evolving role of "the network" in virtualized and cloud computing environments.

The gist of his post, which he backs up with examples from Greg Ness' series on Infrastructure 2.0, is that in order to harness the benefits of virtualization and cloud computing, we must automate; from the endpoint to the underlying platforms -- including the network -- manual processes need to be replaced by automated capabilities:

When was the last time you thought “network” when you heard “cloud computing”? How often have you found yourself working out exactly how you can best utilize network resources in your cloud applications?  Probably never, as to date the network hasn’t registered on most peoples’ cloud radars.

This is understandable, of course, as the early cloud efforts try to push the entire concept of the network into a simple “bandwidth” bucket.  However, is it right? Should the network just play dumb and let all of the intelligence originate at the endpoints?
...
The writing is on the wall. The next frontier to get explored in depth in the cloud world will be the network, and what the network can do to make cloud computing and virtualization easier for you and your organization

If you walked away from James' blog as I did initially, you might be left with the impression that this isn't really about "the network" gaining additional functionality or innovative capabilities, but rather just tarting up the ability to integrate with virtualization platforms and automate it all.

Doesn't really sound all that sexy, does it.  Well, it's really not, which is why even today in non-virtualized environments we don't have very good automation and most processes still come down to Bob at the helpdesk. Virtualization and cloud are simply giving IT a swift kick in the ass to ensure we get a move on to extract as much efficiency and remove as much cost from IT as possible.

Don't be fooled by the simplicity of James' post, however, because there's a huge moose lurking under the table instead of on top of it and it goes toward the fundamental crux of the battle brewing between all those parties interested in becoming your next "datacenter OS" provider.

There exists one catalytic element that produces very divergent perspectives in IT around what, where, why and who automates things and how, and that's the very definition of "the network" in virtualized and cloud models.

How someone might describe "the network" as either just a "bandwidth bucket" of feeds and speeds or an "intelligent, aware, sentient platform for service delivery" depends upon whether you're really talking about "the network" as a subset or a superset of "the infrastructure" at large.

Greg argues that core network services such as IP adddress management, DNS, DHCP, etc. are part of the infrastructure and I agree, but given what we see today, I would say that they are part-in-parcel NOT a component of "the network" -- they're generally separate and run atop the plumbing.  There's interaction, for sure, but one generally relies upon these third party service functions to deliver service.  In fact, that's exactly the sort of thing that Greg's company, Infoblox, sells.

This contributes to part of this definitional quandary.

Now we have this new virtualization layer injected between the network and the rest of the infrastructure which provides a true lever and frictionless capability for some of this automation but further confuses the definition of "the network" since so much of the movement and delivery of information is now done at this layer and it's not integrated with the traditional hardware-based network.*

See what I mean in this post titled "The Network Is the Computer...(Is the Network, Is the Computer...)"

This is exactly why you see Cisco's investment in bringing technologies such as VN-Link and the Nexus-1000v virtual switch to virtualized environments; it homogenizes "the network." It claws back the access layer so they can allow the network teams to manage the network again (and "automate" it) while also getting their hooks deeper into the virtualization layer itself. 

And that's where this gets interesting to me because in order to truly automate virtualized and cloud computing environments, this means one of three things as it relates to where core/critical infrastructure services live:

1. They  will continue to be separate as stand-alone applications/appliances or bundled atop an OS
2. They become absorbed by "the (traditional) network" and extend into the virtualization layer
3. They get delivered as part of the virtualization layer

So if you're like most folks and run Microsoft-based "core network services" for things (at least internally) like DNS, DHCP, etc., what does this mean to you?  Well, either you continue as-is via option #1, you transition to integrated services in "the network" via option #2 or you end up with option #3 by the very virtue that you'll upgrade to Windows Server 2008 and Hyper-V anyway.

SO, this means that the level of integration between, say, Cisco and Microsoft will have to become as strong as it is with VMware in order to support the integration of these services as a "network" function, else they'll continue -- in those environments at least -- as being a "bandwidth bucket" that provides an environment that isn't really automated.

In order to hit the sweet spot here, Cisco (and other network providers) need to then start offering core network services as part of "the network."  This means wrestling it away from the integrated OS solutions or simply buying their way in by acquiring and then integrating these services ($10 Cisco buys Infoblox...)

We also see emerging vendors such as Arista Networks who are entering the grid/utility/cloud computing network market with high density, high-throughput, lower cost "cloud networking" switches that are more about (at least initially) bandwidth bucketing and high-speed interconnects rather than integrated and virtualized core services.  We'll see how the extensibility of Arista's EOS affects this strategy in the long term.

There *is* another option and that's where third party automation, provisioning, and governance suites come in that hope to tame this integration wild west by knitting together this patchwork of solutions. 

What's old is new again.

/Hoff

*It should be noted, however, that not all things can or should be virtualized, so physical non-virtualized components pose another interesting challenge because automating 99% of a complex process isn't a win if the last 1% is a gating function that requires human interaction...you haven't solved the problem, you've just made it less steps that still requires Bob at the helpdesk..