Rational Survivability

IMPORTANT: Moving My Blog & RSS Feed

This is generally a messy thing to do and I'm sure it's going to screw some things up for folks, but I'm moving my blog.  I've decided that after 3 years at TypePad, I need some flexibility to control my own destiny and manage my brand a little better.

This will be my last post here, so please adjust your landing accordingly to now point to:

www.rationalsurvivability.com/blog

If you're using an RSS reader for raw RSS feeds, please adjust your feed to:

feed://www.rationalsurvivability.com/blog/?feed=rss2

If you're using Feedburner, I'm going to adjust the feeds tonight. Hopefully it will work:

Feed Title: Rational Survivability
Feed Address: http://feeds2.feedburner.com/rationalsurvivability/blog

I'm sure this is going to cause pandaemonium, but c'est la vie.

Thanks for your patience.

/Hoff

BeanSec! Wednesday, March 18, 2009 - 6PM to ?

Yo!  BeanSec! is once again upon us.  Wednesday, March 18, 2009.

Middlesex Lounge: 315 Massachusetts Ave, Cambridge 02139. 

BeanSec! is an informal meetup of information security professionals, researchers and academics in the Greater Boston area that meets the third Wednesday of each month.

I say again, BeanSec! is hosted the third Wednesday of every month.  Add it to your calendar.

Come get your grub on and have a drink.  Lots of good people show up.  Really.

Unlike other meetings, you will not be expected to pay dues, “join up”, present a zero-day exploit, or defend your dissertation to attend.

Don't worry about being "late" because most people just show up when they can. 6:30 is a good time to aim for. We'll try and save you a seat. There is a plenty of parking around or take the T.

The food selection is basically high-end finger-food appetizers and the drinks are really good; an attentive staff and eclectic clientèle make the joint fun for people watching. Zach and I will generally annoy you into participating somehow, even if it's just fetching napkins. ;)

This week's BeanSec refreshments sponsored by: IOActive

We often retire across the street to Asgard for more substantive fare after the event and then to Tosci's for coffee...

A little administrivia note: After 2 years, we're finally getting the beansec.org domain, blog, email, etc. setup...expect completion in about a week.

See you there!

/Hoff

How To Be PCI Compliant in the Cloud...

I kicked off a bit of a dust storm some months ago when I wrote a tongue-in-cheek post titled "Please Help Me: I Need a QSA to Assess PCI/DSS Compliance In the Cloud."  It may have been a little contrived, but it asked some really important questions and started some really good conversations on my blog and elsewhere.

At SourceBoston I sat in on Mike Dahn's presentation titled "Cloud Compliance and Privacy" in which he did an excellent job outlining the many issues surrounding PCI and Compliance and it's relevance to Cloud Computing.  

Shortly thereafter, I was speaking to Geva Perry and James Urquhart on their "Overcast" podcast and the topic of PCI and Cloud came up. 

Geva asked me if after my rant on PCI and Cloud if what I was saying was that one could never be PCI compliant in the Cloud.  I basically answered that one could be PCI compliant in the Cloud depending upon the services used/offered by the provider and what sort of data you trafficked in.

Specifically, Geva made reference to the latest announcement by Rackspace regarding their Mosso Cloud offering and PCI compliance in which they tout that by using Mosso, a customer can be "PCI Compliant"  Since I hadn't seen the specifics of the offering, I deferred my commentary but here's what I found:

Cloud Sites, Mosso|The Rackspace Cloud’s Flagship offering, is officially the very first cloud hosting solution to enable an Internet merchant to pass PCI Compliance scans for both McAfee’s PCI scans and McAfee Secure Site scans. 

This achievement occurred just after Computer World published an article where some CIO’s shared their concern that Cloud Computing is still limited to “things that don’t require full levels of security.”  This landmark breakthrough may be the beginning of an answer to those fears, as Mosso leads Cloud Hosting towards a solid future of trust and reliability.

Mosso's blog featured an example of a customer -- The Spreadsheet Store -- who allegedly attained PCI compliance by using Mosso's offering. Pay very close attention to the bits below:

“We are making the Cloud business-ready.  Online merchants, like The Spreadsheet Store can now benefit from the scalability of the Cloud without compromising the security of online transactions,” says Emil Sayegh, General Manager of Mosso|The Rackspace Cloud.  “We are thrilled to have worked with The Spreadsheet Store to prepare the Cloud for their online transactions.”

...

The Spreadsheet Store set up their site using aspdotnetstorefront, “Which is, in our opinion, the best shopping cart solution on the market today,” says Murphy.  “It also happens to be fully compatible with Mosso.”  Using Authorize.Net, a secure payment gateway, to handle credit card transaction, The Spreadsheet Store does not store any credit card information on the servers.  Murphy and team use MaxMind for fraud prevention, Cardinal Commerce for MasterCard Secure Code and Verified by Visa, McAfee for PCI and daily vulnerability scans, and Thawte for SSL certification.

So after all of those lofty words relating to "...preparing the Cloud for...online transactions," what you can decipher is that Mosso doesn't seem to provide services to The Spreadsheet Store which are actually in scope for PCI in the first place!*

The Spreadsheet store redirects that functionality to a third party card processor!  

So what this really means is if you utilize a Cloud based offering and don't traffic in data that is within PCI scope and instead re-direct/use someone else's service to process and store credit card data, then it's much easier to become PCI compliant.  Um, duh. 

The goofiest bit here is that in Mosso's own "PCI How-To" (warning: PDF) primer, they basically establish that you cannot be PCI compliant by using them if you traffic in credit card information:

Cloud Sites is not currently designed for the storage or archival of credit card information.  In order to build a PCI compliant e-commerce solution, Cloud Sites needs to be paired up with a payment gateway partner.

Doh!

I actually wrote quite a detailed breakdown of this announcement for this post yesterday, but I awoke to find my buddy Craig Balding had already done a stellar job of that (curses, timezones!)  I'll refer you to his post on the matter, but here's the gem in all of this.  Craig summed it up perfectly:

The fact that Mosso is seeking ways to help their customers off-load as much PCI compliance requirements to other 3rd parties is fine - it makes business sense for them and their merchant customers.  It’s their positioning of the effort as a “landmark breakthrough” and that they are somehow pioneers which leads to generalisations rooted in misunderstandings that is the problem.
Next time you hear someone say ‘Cloud Provider X is PCI compliant’, ask the golden PCI question: is their Cloud receiving, processing, storing or transmitting Credit Card data (as defined by the PCI DSS)?  If they say ‘No’, you’ll know what that really means…marketecture.

There's some nifty marketing for you, eh?

--

* Except for the fact that the web servers housed at Mosso must undergo regularly-scheduled vulnerability scans -- which Mosso doesn't do, either.

On the Overcast Podcast with Geva Perry and James Urquhart

Geva and James were kind (foolish?) enough to invite me onto their Overcast podcast today:

In this podcast we talk to Christopher Hoff, renowned information security expert, and especially security in the context of virtualization and cloud computing. Chris is the author of the Rational Survivability blog, and can be followed as @Beaker on Twitter.
Show Notes:

• Chris talks about some of the myths and misconceptions about security in the cloud. He addresses the claim that Cloud Providers Are Better At Securing Your Data Than You Are and the benefits and shortcomings of security in the cloud.
• We talk about Chris's Taxonomy of Cloud Computing (excuse me, model of cloud computing)
• Chris goes through some specific challenges and solutions for PCI-compliance in the cloud
• Chris examines some of the security issues associated with multi-tenant architecture and virtualization

Check it out here.

/Hoff 

More On Clouds & Botnets: MeatClouds, CloudFlux, LeapFrog, EDoS and More!

After my "Frogs" talk at Source Boston yesterday, Adam O'Donnell and I chatted about one of my chuckle slides I threw up in the presentation in which I give some new names to some (perhaps not new) attack/threat scenarios which involve Cloud Computing:

• MeatCloud - Essentially abusing Amazon's Mechanical Turk and using it to produce the Cloud version of a sweat shop; exploiting the ignorant for fun and profit to perform menial illegal muling tasks on your behalf...think SETI meets underage garment workers...
• CloudFlux - Take a mess of stolen credit cards, open up  a slew of Amazon AWS accounts using them, build/scale to thousands of instances overnight, launch carpet bomb attack (you choose,) tear it down/have it torn down, and move your botnet elsewhere...rinse, lather, repeat...
• LeapFrog - As we move to hybrid private/public clouds and load balancing/cloudbursting across multiple cloud providers, we'll interconnect Clouds via VPNs to the "trusted internals" of your Cloudbase... Attackers will thank us by abusing these tunnels to penetrate your assets through the, uh, back door.
• vMotion Poison Potion - When VMware's vCloud makes its appearance and we start to allow vMotion across datacenters and across Clouds (in the clear?,) imagine the fun we'll have as we see attacks against vMotion protocols and VM state...  
• EDoS - Economic Denial of Sustainability - Covered previously here. 

Adam mentioned that I might have considered that Botnets were a great example of a Cloud-based service and wrote a very cool piece about it on ZDNet here.

I remembered after the fact that I wrote a related blog on the topic several months ago titled "Cloud Computing: Invented by Criminals, Secured by ???" as a rif on something Reuven Cohen wrote.

/Hoff

Source Boston - Video Interviews of Security Rockstars...

Source Boston has officially wound down, but I'm still on Cloud 9 (sorry) following the amazing sessions and interaction I had with my fellow attendees and speakers.

My presentation was well received and with Marcus Ranum, Dan Geer, and Adam Shostack sitting six feet in front of me, I didn't choke as badly as I could have.  I had a ton of fun giving this first run preso and got a lot of great feedback and questions.

One of the most excellent things I got to do was spend some time walking about with Zach Lanier (@quine on Twitter) and interview many of the vendors and speakers extemporaneously on various subjects.

I'll be updating this post with links to the interviews as I get them cleaned up and uploaded to YouTube...

Here's a sampling of what you can expect:

• David Mortman, "I Can Haz Privacy"
• Dmitry McKay, LogLogic
• Chris Wysopal - Veracode 
• Peter Kuper - "Silver Linings"
• Jose Nazario, Arbor, "Politically Motivated DDoS Attacks" 
• Christien Rioux, Source
• Jeremiah Grossman, Whitehat Security, "Get Rich or Die Trying, Making Money the Black Hat Way"
• Amrit Williams, BigFix, "The Economics of CyberCrime & the Law of Malware Probability"  
• Adam Shostack, Microsoft, "The Crisis In Information Security"  
• Dan Kaminsky, IOActive, "DNS - Toward a Secure Infrastructure" 
• Chris Weber, Casaba Security, "Exploiting Unicode-Enabled Software"  
• Rob Cheyne, SafeLight, "The End Of Our Rope: The On-Going Discussion Between Business & Security" 

You'll laugh, you'll cry, you'll wonder why people gave me this task...

But seriously, we discuss such mega-issues such as DDoS, Snuggies, Bedazzlers, Zombies and Estonian dissident groups (and that's in just ONE of the talks.)  

I think I've found something I absolutely LOVE doing -- vlogging (video blogging) and will try and do more of it.

Check back for updates to the links over the weekend.

/Hoff

Oh Noes: We Can't Monitor/Protect Against Intra-VM Traffic!

I got a press release in my inbox this morning that made me cringe.  It came from a vendor who produces a "purpose-built virtual firewall."

The press release details a customer case study that I found typical of how security solutions are being marketed in the virtualization space today, which again is really more about visibility than pure "security" and preys mostly on poor planning and fundamental issues stemming from treating "security" like a band-aid instead of an element of enterprise architecture.

When we start to cross the streams as to the realities of virtualization, the security implications thereof and making promises to solve problems with products which may or may not be deserving of investment given an assessment of risk, especially in today's trying economic climate, it makes me cranky.

I'm just tiring of the mixing of metaphors in the marketing of these "solutions."

I was specifically annoyed by a couple of statements in the press release and since I haven't had my coffee, I thought I'd point out a few to further underscore what I present in my Four Horsemen presentation regarding where we are in the solution continuum today.

To wit:

[Customer] has selected the [Vendor's] virtual firewall to secure its virtual environment and mitigate an attack before it could hit their network. 

Given the fact that to get to a VM you generally have to (1) utilize the physical network and (2) transit the vSwitch in the VMM, the reality is that an attack has already "hit their network" long before it gets to the VM or the virtual firewall, at least given today's available offerings.  There is no magic security fairy dust that will mitigate an attack presciently.

If you put VM's into production that are already infected, you have other problems to solve...

“After moving our production applications to a virtualized environment we realized that we lacked security; I had no visibility into what was going on between VMs and a virtual attack could take down our network,” said [Customer.]  “We sought the same level of security for our virtual environment that we had with our physical network.”

This indicates a lack of proper risk management and planning on the part of the [Customer.]  Further, it underscores an example I use in the Four Horsemen which concerns which tools in a multi-server physical deployment did the [Customer] use to monitor/protect in-line traffic between these physical machines? 

The {Customer] must have done this since the press release suggests they demand the "...same level of security for [their] virtual environment that [they] had with [their] physical network."

Did the [Customer] have each physical server on it's own VLAN/subnet, isolated with firewalls?  Did he SPAN every single port to an IDS/IPS?  If not, what's the difference here?  The Hypervisor?  What protection mechanisms has the fancy virtual firewall put in place to protect it?  None.

[Customer] was increasingly concerned about the risks of virtual networks, which range from security policy violations such as mixing trusted and un-trusted systems to malware exploits that can propagate undetected within a virtual network. 

Based upon the second paragraph above where the [Customer] admitted they put their virtualized environment into production without visibility or security, they clearly weren't that concerned with the risks.

A large amount of data center network traffic was moving between VMs and [Customer] had no visibility or control over the communication on the virtual network.

So if there were no security or visibility tools in place, how was it determined that traffic was moving between VM's?

Does this mean that all the customer's VM's were in a single VLAN and not segmented? If not and vSwitch configurations via port groups and VLANs were configured around VM criticality, role or function, then they certainly had some insight into what was moving between VM's and the "data center," right?

I must be confused. 

[Customer's] traditional network security tools could not monitor, analyze or troubleshoot inter-VM traffic because communications between VMs on the same physical host never touch the traditional network. 

Assuming that the VM's weren't in a single VLAN/portgroup on a single vSwitch and instead were segmented via VLANs/subnets, then the only way to get traffic from VLAN/IP Subnet A to VLAN/IP Subnet B (and thus VM A to VM B in these VLAN's) is though a layer 3 routing process which generally means traffic exits the virtual network and hits the physical network...where said "traditional security tools" could see it.

Of course, this doesn't help intra-VM traffic on the same portgroup/VLAN/vSwitch, but that's not what they pointed to above, but assuming they don't look at inter-machine traffic in their physical network on the same VLAN, again I ask what's the difference?

VMs were able to communicate with each other without observation or policy-based inspection and filtering, which left them highly vulnerable to malicious exploits. Additionally, worms and viruses could further spread among physical hosts via unintentional VMotion of an infected VM.

Back to my point above about how the [Customer] monitored traffic between physical hosts...if you don't do it in physical environments, why the fret in the virtual?

Oh and "unintentional VMotion!?" ZOMG!  For a VM to be "infected," excluding direct physical access, wouldn't the threat vector be the network in the first place?  

The [Vendor] virtual firewall was specifically created to mitigate the risks of virtual networks, while maintaining the ROI of virtualization.

What "risks of virtual networks" does this product mitigate in the absence of vulnerability or clearly defined threats that aren't faced in the physical realm?  Let me tell you.  It goes back to the very valid claim that you get better visibility given the integration with the virtualization platform configuration managers to call attention to when CHANGE occurs.

This is the real value of products like this from [Vendor.]  In the long run, the big boys who make mature firewalls and IPS products will get to harness API's like VMsafe and combined with the compartmentalization and segmentation capabilities of vShield Zones leaves a very short runway for products like this.

I'm not suggesting that products like this from [Vendor] don't offer value and solve an immediate pain point. I'd even consider deploying them to solve very specific problems I might have, but then again, I know what problem I'd be trying to solve. ROI?  Oy.

However, unlike the picture painted of the [Customer] above, I plan a little better and understand the impact virtualization has on my security posture and how that factors into my assessment and management of risk BEFORE I put it into production.  You should too.

</rant>

{Ed: I use 'intra-' instead of 'inter-' to reflect the "internal" passing of traffic between VM's using the vSwitch. Should traffic exit the vSwitch/host and hit the network as part of interchange between two VM's, I'd count this as 'inter-" VM traffic.}

Sun vs. Cisco? I'm Getting My Popcorn...

Scott Lowe wrote an interesting blog today wondering if Sun was preparing to take on Cisco in the virtualization space, referencing the development of virtualized networking functionality featuring the novel combination of commodity hardware and open source software to unseat the Jolly Green Giant:

A while back in Virtualization Short Take #25 I briefly mentioned Sun’s Crossbow network virtualization software, which brings new possibilities to the Solaris networking world. Not being a Solaris expert, it was hard for me at the time to really understand why Solaris fans were so excited about it; since then, though, I’ve come to understand that Crossbow brings to Solaris the same kind of full-blown virtual network interfaces and such that I use daily with VMware ESX. Now I’m beginning to understand why people are so thrilled!

In any case, an astute reader picked up on my mention of Crossbow and pointed me to this article by Jonathan Schwartz of Sun, and in particular this phrase:

You’re going to see an accelerating series of announcements over the coming year, from amplifying our open source storage offerings, to building out an equivalent portfolio of products in the networking space…

That seemingly innocuous mention was then coupled with this blog post and the result was this question: is Sun preparing to take on Cisco? Is Sun getting ready to try to use commodity hardware and open source software to penetrate the networking market in the same way that they are using commodity hardware and open source software to try to further penetrate the storage market with their open storage products (in particular, the 7000 series)?

It’s an interesting thought, to say the least. Going up against Cisco is a bold move, though, and I question Sun’s staying power in that sort of battle. Of course, with Cisco potentially distracted by the swirling rumors regarding the networking giant’s entry into the server market, now may be the best time to make this move.

It's really the last paragraph that is of interest to me, specifically the boldfaced sentence I highlighted.  I think the "rumors" have pretty much been substantiated by the mainstream press, so let's assume "California" is going to happen.

Let's make a couple of things really, really clear:

1. I don't know how anyone can think that Cisco is "distracted" by bringing to market the logical extension of virtualized infrastructure -- the compute function -- as anything other than a shrewd business decision to offer a complete end-to-end solution to customers.  I talked about it here in blog post titled "Cisco Is NOT Getting Into the Server Business..." This is an Enterprise Architecture play, pure and simple.
2. Honestly, if we're discussing commoditization, a server is a server is a server, whether it's in a blade form factor or not, and it's not like Cisco has to worry about building things from scratch. The availability of OEM/ODM components (raw or otherwise) means they don't have to start from scratch.  Oh yes, I know HP spent a bazillion dollars on C-Class fan engineering and IBM's BCHT is teh awesome and...
3. The whole game is Unified Computing; bringing together enterprise class compute, network and storage as a solution with integrated virtualization, management and intelligence; you take the biggest pain point out of the equation -- integration -- and you drive down cost while increasing utility, agility and efficiency.
4. If you look at what "California" is slated to deliver it's hard to see how Sun would compete: A blade based chassis with integrated Nexus converged networking/storage, integrated virtualization from VMware (with Nexus/VN-Link,) and management from BMC.  You know, Enterprise stuff, not integration hodge podge. 

So, I ask, does this look like a distraction to you? 

I'm not knocking Sun (or Scott to be clear,) but if I were they, I'd be much more worried about HP or IBM or even Microsoft and Redhat.

I'm grabbing my popcorn, but this battle might be over before the kernels (ha!) start popping.

/Hoff

Cloud Computing Not Ready For Prime Time?

I just read another in a never-ending series of articles that takes a polarized view of Cloud Computing and its readiness for critical applications and data.

In the ComputerWorld article titled "Cloud computing not ready for critical apps,", Craig Steadman and Patrick Thibodeau present some very telling quotes from CIO's of some large enterprises regarding their reticence toward utilizing "Cloud Computing" and it's readiness for their mission critical needs.

The reasons are actually quite compelling, and I speak to them (and more) in my latest Cloud Computing presentation which I am giving at Source Boston this week:

Reliability, availability and manageability are all potential show-stoppers for the CIO's in this article, but these are issues of economic and adoptive context that don't present the entire picture. 

What do I mean?

At the New England Cloud Computing Users' Group, a Cloud-based startup called Pixily presented on their use of Amazon's AWS services. They painted an eye-opening business case which detailed the agility and tremendous cost savings that the "Cloud" offers.  "The Cloud" provides them with reduced time-to-market, no up-front capital expenditures and allows them to focus on their core competencies. 

All awesome stuff.

I asked them about how their use of AWS and what amounted to a sole-source service provider did to their disaster recovery, redundancy/resiliency and risk management processes.  They had to admit that the day they went live with feature coverage on the front page of several newspapers also happened to be the day that Amazon suffered an 8 hour outage, and thus, so did they.

Now, for a startup, the benefits often outweigh the risks associated for downtime and vendor lock-in. For an established enterprise with cutthroat service levels, regulatory pressures and demanding customers who won't/can't tolerate outages, this is not the case.

Today we're suffering from issues surrounding the fact that emerging offerings in Cloud Computing are simply not mature if what you're looking for involves the holistic and cohesive management, reliability, resilience and transparency across suppliers of Cloud services.

We will get there as adoption increases and businesses start to lean on providers to create and adopt standards that answer the issues above, but today if you're an enterprise who needs five 9's, you may come to the same conclusion as the CIO's in the CW article.  If you're an SME/SMB/Startup, you may find everything you need in the Cloud.

It's important, however, to keep a balanced, realistic and contextual perspective when addressing Cloud Computing and its readiness -- and yours -- for critical applications.  Polarizing the discussion to one hyperbolic end or the other is not really helpful.

/Hoff

If Virtualization is a Religion, Does That Make Cloud a Cult?

I had just finished reading Virtual Gipsy's post titled "VMware as religion" when my RSS reader featured a referential post from VM/ETC's Rich titled "vTheology: the study of virtualization as religion."

While I appreciated the humor surrounding the topic, I try never to mix friends politics, and religion* so I'll not wade into the deep end on this one except to suggest what my title asks: 

If virtualization is a religion, does that make cloud a cult?

If so, to whom do I send my tidings?  Who is the Cardinal of the Cloud?  The Pope of PaaS?  The Shaman of Service?

...and where's my phenobarbital, vodka and purple cape?

/Hoff

*...and truth be told, I'm not feeling particularly witty this morning.

Incomplete Thought: Offensive Computing - The Empire Strikes Back

Yesterday at IANS, Greg Shipley gave a great keynote that focused on a lot of things we do today in InfoSec that aren't necessarily as effective as they should be. Greg called for a change in our behavior as a community to address the gaps we have.

In the Q&A section, it occurred to me that for the sake of argument, I would ask Greg about his thoughts on changing our behavior and position in dealing with security and our adversaries by positing that instead of always playing defense, we should play some offense.

I didn't constrain what I meant by "offense" other to suggest that it could include "active countermeasures," but what is obvious is that people immediately throw up walls around being "offensive" without spending much time defining what it actually means.

I've written and spoken about this before, but it's a rather contentious issue. It gets shelved pretty quickly by most but it really shouldn't in my opinion.

In a follow-on discussion after the keynote, Marcus Ranum, Richard Bejtlich, Rocky DeStefano and I were standing around shooting the, uh, stuff, when I brought this up again.

We had a really interesting dialog wherein we explored what "offensive computing" meant to each of us and it was clear that simply playing defense alone would never allow us to do anything more than spend money and hope.

There's not been a war yet that has been won with defense alone, so why do we expect we can win this one by simply piling on more barbed wire when the enemy is dropping smart bombs? This is the definition of insanity and a behavior that we don't talk about changing.

"Don't spend money on AV because it's not effective" is an interesting behavioral change from the perspective of how you invest. Don't lay down and take it up the assets by only playing defense is another.

I'm being intentionally vague, obtuse and non-specific when it comes to defining what I mean by "offensive," but we're at a point in time where at a minimum we have the technology and capability to add a little "offense" to our defense.  

You want a change in behavior?  How about not playing the victim?

What are your thoughts on "offensive computing?"  

/Hoff