Rational Survivability

HyperJackStacking? Layers of Chewy VMM Goodness -- the BLT of Security Models

So Mogull is back on the bench and I'm glad to see him blogging again. 

As I type this, I'm listening to James Blunt's  new single "1973" which is unfortunately where Rich's timing seems to be on this topic.  'Salright though.  Can't blame him.  He's been out scouting the minors for a while, so being late to practice is nothing to be too wound up about.

<If you can't tell, I'm being sarcastic.  I only wish that Rich was when he told me that his favorite TexMex place in his hometown is called the "Pink Taco."  That's all I'm going to say about that...>

The notion of the HyperJackStack (Hypervisor Jacking & Stacking) is actually a problem set that has been discussed at length and in the continuum of these discussions happened quite a while ago. 

To put it bluntly, I believe the discussion -- for right or wrong -- stepped over this naughty little topic months ago in lieu of working from the bottom up for the purpose exposing fundamental architectural deficiencies (or at least their potential) in the core of virtualization technology.  This is an argument that parallels dissecting a BLT sandwich...you're approaching getting to the center of a symmetric stack so which end you start at is almost irrelevant.

The good/bad VMM/HV problem has really been relegated to push-pin on the to-do board of all of the virtualization vendors and this particular problem has been framed by said vendors to be apparently solved first operationally from the management plane and THEN dealt with from the security perspective.

So Rich argues that after boning up on Joanna and Thom's research that they're arguing the wrong case completely for the dangers of virtualized rootkits.  Instead of worrying about undetectability of this or that -- pills and poultry be damned -- one should be focused on establishing the relative disposition of *any* VMM/Hypervisor running in/on a host:

Problem is, they’re looking at the wrong problem. I will easily concede that detecting virtualization is always possible, but that’s not the real problem. Long-term virtualization will be normal, not an exception, so detecting if you’re virtualized won’t buy you anything. The bigger problem is detecting a malicious hypervisor, either the main hypervisor or maybe some wacky new malicious hypervisor layered on top of the trusted hypervisor.

To Rich's credit, I think that this is a huge problem and one that deserves to be solved.  That does not mean that I think one is the "right" versus "wrong" problem to solve, however.  Nor does it mean this hasn't been discussed.  I've talked about it many times already.  Maybe not as eloquently...

The flexibility of virtualization is what provides the surface expansion of vectors for threat; you can spin up, move or kill a VM across an enterprise with a point-click.  So the first thing to do before trying to determine if a VMM/HV is malicious is to detect its presence and layering in the first place...this is where Thom/Joanna's research really does make sense.

You're approaching this from a different direction, is all.

Thom responded here, and I have to agree with his overall posture; the notion of putting hooks into the VMM/HV to allow for "external" detection mechanisms for the sake solely of VMM/HV rootkit detection is unlikely given the threat, but we are already witness to the engineered capacities to allow for "plug-ins" such as Blue Lane's that function "along side" the HV/VMM and there's nothing saying one couldn't adapt a similar function for this sort of detection (and/or prevention) as a value-add.

Ultimately though, I think that the point of response boils down to the definition of the mechanisms used in the detection of a malicious VMM/HV.  I ask you Rich, please define a "malicious" VMM/HV from one steeped in goodness. 

This sounds like in practice, it will come down to yet another iteration of the signature-driven IPS circle jerk to fingerprint/profile disposition.  We'll no doubt see anomaly and behavioral analysis used here, and then we'll have hashing, memory firewalls, etc...it's going to be the Hamster Wheel all over again.  For the same reason we have trouble with validating security and compliance state for anything more than the cursory checks @ 30K feet today, you'll face the same issue with virtualization -- only worse.

I've got one for you...how about escaping from the entire VM "jail" entirely...Ed Skoudis over @ IntelGuardians just did an interview with the PaulDotCom boys on this topic...

I believe one must start from the bottom and work up; they're trying to make up for the fact that this stuff wasn't properly thought through in this iteration and are trying to expose the issue now. In fact, look at what Intel just announced today with vPro:

New in this product is Intel Trusted Execution Technology (Intel TXT, formerly codenamed LaGrande). Intel TXT protects data within virtualized computing environments, an important feature as IT managers are considering the adoption of new virtualization-enabled computer uses. Used in conjunction with a new generation of the company's virtualization technology - Intel Virtualization Technology for Directed I/O - Intel TXT ensures that virtual machine monitors are less vulnerable to attacks that cannot be detected by today's conventional software-security solutions. By isolating assigned memory through this hardware-based protection, it keeps data in each virtual partition protected from unauthorized access from software in another partition.

So no, Ptacek and Joanna aren't fighting the "wrong" battle, they're just fighting one that garners much more attention, notoriety, and terms like "HyperJackStack" than the one you're singling out.  ;)

/Hoff

P.S. Please invest in a better setup for your blog...I can't trackback to you (you need Halo or something) and your comment system requires registration...bah!  Those G-Boys have you programmed... ;)

For Sale / Special Price: One (Un)detectable Hyperjacking PillWare: $416,000. Call Now While Supplies Last!

Joanna Rutkowska of "Invisible Things" Blue Pill Hypervisor rootkit fame has a problem.  It's about 6 foot+ something, dresses in all black and knows how to throw down both in prose and in practice.

Joanna and crew maintain that they have the roughed-out prototype that supports their assertion that their HyperJacking malware is undetectable.  Ptacek and his merry band of Exploit-illuminati find this a hard pill to swallow and reckon they have a detector that can detect the "undetectable."

They intend to prove it.  This is awesome!  It's like the Jackson/Lidell UFC fight.  You don't really know who to "root" for, you just want to be witness to the ensuing carnage!

We've got a stare down.  Ptacek and crew have issued a challenge that they expect -- with or without Joanna's participation -- to demonstrate successfully at BlackHat Vegas:

Joanna, we respectfully request terms under which you’d agree to an “undetectable rootkit detection challenge”. We’ll concede almost anything reasonable; we want the same access to the (possibly-)infected machine than any antivirus software would get.

The backstory:

• Dino Dai Zovi, under Matasano colors, presented a hypervisor rootkit (“Vitriol”) for Intel’s VT-X extensions at Black Hat last year, at the same time as Joanna presented BluePill for AMD’d SVM.

• We concede: Joanna’s rootkit is coolor than ours. I particularly liked using the debug registers to grab network traffic out of the drivers. We stopped weaponizing Vitriol.

• Peter Ferrie, the Symantec branch of our Black Hat team, releases a kick-ass paper on hypervisor detection. Peter’s focus is on fingerprinting software hypervisors (like VMWare), but he also comes up with a clever way to detect hardware virtualization.

• Nate Lawson, Dino, and I are, simultaneously, working on hardware rootkit detection techniques.

• Nate, Peter, Dino, and I join up to defend our thesis at Black Hat: if you surreptitiously “hyperjack” an OS, enabling hardware virtualization (or replacing or infecting an existing hypervisor), you introduce so many subtle changes in system behavior —- timing and otherwise —- that you’re bound to be detectable.

...and Joanna respondeth, signaling her "readiness" and conditions for the acceptance of said challenge:

Thomas Ptacek and company just came up with this funny challenge to test our Blue Pill rootkit. And, needles to say, the Invisible Things Lab team is ready to take their challenge, however with some additional requirements, that would assure the fairness of the contest.

First, we believe that 2 machines are definitely not enough, because the chance of correct guess, using a completely random (read: unreliable) detection method is 50%. Thus we think that the reasonable number is 5 machines. Each of them could be in a state 0 or 1bluepill.exe and bluepill.sys

The .sys file is digitally signed, so it loads without any problem (we could use one of our methods for loading unsigned code on vista that we're planning to demonstrate at BH, but this is not part of the challenge, so we will use the official way).

The bluepill.exe takes one argument which is 0 or 1. If it's 1 it loads the driver and infects the machines. If it's 0 it also loads the driver, but the driver does not infect the machine.

So, on each of the 5 machines we run bluepill.exe with randomly chosen argument, being 0 or 1. We make sure that at least one machine is not infected and that at least one machine is infected.

After that the detection team runs their detector.exe executable on each machine. This program can not take any arguments and must return only one value: 0 or 1. It must act autonomously -- no human assistance when interpreting the results.

The goal of the detection team is to correctly mark each machine as either being infected (1) or not (0). The chance of a blind guess is:

(i.e. infected or not). On each of this machines we install two files:

1/(2^5-2) = 3%

The detector can not cause system crash or halt the machine -- if it does they lose. The detector can not consume significant amount of CPU time (say > 90%) for more then, say 1 sec. If it does, then it's considered disturbing for the user and thus unpractical.

The source code of our rootkit as well as the detector should be provided to the judges at the beginning of the contests. The judges will compile the rootkit and the detector and will copy the resulting binaries to all test machines.

After the completion of the contest, regardless of who wins, the sources for both the rootkit and the detector will be published in the Internet -- for educational purpose to allow others to research this subject.

Our current Blue Pill has been in the development for only about 2 months (please note that we do not have rights to use the previous version developed for COSEINC) and it is more of a prototype, with primary use for our training in Vegas, rather then a "commercial grade rootkit". Obviously we will be discussing all the limitations of this prototype during our training. We believe that we would need about 6 months full-time work by 2 people to turn it into such a commercial grade creature that would win the contest described above. We're ready to do this, but we expect that somebody compensate us for the time spent on this work. We would expect an industry standard fee for this work, which we estimate to be $200 USD per hour per person.

If Thomas Ptacek and his colleges are so certain that they found a panacea for virtualization based malware, then I'm sure that they will be able to find sponsors willing to financially support this challenge.

As a side note, the description for our new talk for Black Hat Vegas has just been published yesterday.

So, if you get past the polynomial math, the boolean logic expressions, and the fact that she considers this challenge "funny," reading between the HyperLines, you'll extract the following:

1. The Invisible Things team has asserted for some time that their rootkit is 100% undetectable
2. They've worked for quite sometime on their prototype, however it's not "commercial grade"
3. In order to ensure success in winning the competition and thus proving the assertion, they need to invest time in polishing the rootkit
4. They need 5 laptops to statistically smooth the curve
5. The Detector can't impact performance of the test subjects
6. All works will be Open Sourced at the conclusion of the challenge
   (Perhaps Alan Shimel can help here! ;) ) and, oh, yeah...
7. They have no problem doing this, but someone needs to come up with $416,000 to subsidize the effort to prove what has already been promoted as fact

That last requirement is, um, unique.

Nate Lawson, one of the challengers, is less than impressed with this codicil and respectfully summarizes:

The final requirement is not surprising. She claims she has put four person-months work into the current Blue Pill and it would require twelve more person-months for her to be confident she could win the challenge. Additionally, she has all the experience of developing Blue Pill for the entire previous year.

We’ve put about one person-month into our detector software and have not been paid a cent to work on it. However, we’re confident even this minimal detector can succeed, hence the challenge. Our Blackhat talk will describe the fundamental principles that give the detector the advantage.

If Joanna’s time estimate is correct, it’s about 16 times harder to build a hypervisor rootkit than to detect it. I’d say that supports our findings.

I'm not really too clear on Nate's last sentence as I didn't major in logic in high school, but to be fair, this doesn't actually discredit Joanna's assertion; she didn't say it wasn't difficult to detect HV rootkits, she said it was impossible. Effort and possibility are mutually exclusive.

This is going to be fun.  Can't wait to see it @ BlackHat.

See you there!

/Hoff

The Philosophy of Network Security Design

Thomas and I were barking at each other regarding something last night and today he left a salient and thought-provoking comment that provided a very concise, pragmatic and objective summation of the embedded vs. overlay security quagmire:

     "I think the jury is still out on how much security policy we   
     should be pushing to middleboxes, and how smart those   
     middleboxes should be. What I know right now is we spend
     way, way too much time, effort, and money on 19" rack
     mountable chasses that suck in packets and spit them back
     out again without providing any measurable impact on the
     security of our networks.  Not a fan."

I couldn't agree more.  Most of the security components today, including those that run in our little security ecosystem, really don't intercommunicate.  There is no shared understanding of telemetry or instrumentation and there's certainly little or no correlation of threats, vulnerabilities, risk or disposition.

The problem is bad inasmuch as even best-of-breed solutions usually require box sprawl and stacking and don't necessarily provide for a more secure posture, especially within context of another of Thomas' interesting posts on defense in depth/mesh...

That's changing, however.  Our latest generation of NPMs (Network Processing Modules) allow discrete security ISV's (which run on intelligently load-balanced Application Processor Modules -- Intel blades in the same chassis) to interact with and control the network hardware through defined API's -- this provides the first step in that common telemetry such that while application A doesn't need to know about the specifics of application B, they can functionally interact based upon the common output of disposition and/or classification of flows between them.

Later, they'll be able to perhaps control each other through the same set of API's.

So, I don't think we're going to solve the interoperability issue completely anytime soon inasmuch as we'll go from 0 to 100%, but I think that the consolidation of these functions into smaller footprints that allow for intelligent traffic classification and disposition is a first good step.

I don't expect Thomas to agree or even resonate with my statements below, but I found his explanation of the problem space to be dead on.  Here's my explanation of an incremental step towards solving some of the bigger classes of problems in that space which I believe hinges on consolidation of security functionality first and foremost.

The three options for reducing this footprint are as follows:

1. Proprietary Embedded security in routers/switches (Cisco, Juniper)
   
   Pros: Supposedly less boxes, better communication between components and good coverage
   given the fact that the security stuff is in the infrastructure.  One vendor from which you get
   your infrastructure and your protection.  Correlation across the network "fabric" will ultimately
   allow for near-time zoning and quarantine.  Single management pane across the Enterprise
   for availability and security.  Did I mention the platform is already there?
   
   Cons: You rely on a single vendor's version of the truth and you get closer to a monoculture
   wherein the safeguards protecting the network put at risk the very assets they seek to protect
   because there is no separation of "church and state."  Also, the expertise and coverage as well
   as the agility for product development based upon evolving threats is hampered by the many
   moving parts in this machine.  Utility vs Security?  Utility wins.  Good enough vs. Best of breed?
   Probably somewhere in between.

 

2. Proprietary Overlay security in a Consolidated Platform (Fortinet 5000, Tipping Point, etc.)
   
   Pros:  Reduced footprint, consolidated functionality, single management pane across multiple
   security functions within the box.  Usually excels in one specific area like AV and can add "good enough" functionality as the needs arise.  Software moves up and down the scalability stack depending upon performance needed.
   
   Cons:  You again rely on a single vendor's version of the truth.  These boxes tend to want to replace switching infrastructure.  Many of these platforms utilize ASICs to accelerate certain functions with the bulk of functionality residing in pure software with limited application or network-level intelligence.  You pay the price in terms of performance and scale given the architectures of these boxes which do not easily allow for the addition of new classes of solutions to thwart new threats.  Not really routers/switches.

 

3. Open Overlay security in a Consolidated Platform (Crossbeam)
   
   Pros:  The customer defines best of breed and can rapidly add new security functionality
   at a speed that keeps pace with the threats the customer needs to mitigate.  Utilizing a scalable and high-performance switching architecture combined with all the benefits
   of an open blade-based security application/appliance delivery mechanism gives the best of all
   worlds: self-healing, highly resilient, high performance and highly-available while utilizing
   hardened Linux OS across load-balanced, virtualized security applications running on optimized
   hardware.
   
   Cons: Currently based upon proprietary (even though Intel reference design) hardware for
   the application processing while also utilizing proprietary networking switching fabric and
   load balancing.  Can only offer software as quickly as it can be adapted and tested on the
   platforms.  No ASICs means small packet performance @ 64byte zero loss isn't as high as
   ASIC based packet-forwarding engines.  No single pane of management.

I think that option #3 is a damned good start towards solving the consolidation issues whilst balancing the need to overlay syngergistically with the network infrastructure.  You're not locked into single vendor's version of the truth and although the hardware may be "proprietary," the operating system and choice in software is not.  You can choose from COTS, Open Source or write your own, all in an scaleable platform that is just as much a collapsed switching/routing platform as it is a consolidated blade server.

I think it has the best chance of evolving to solve more classes of problems than the other two at a rate and level of cost-effectiveness balanced with higher efficacy due to best of breed.

This, of course, depends upon how high the level of integration is between the apps -- or at least their dispositions.  We're working very, very hard on that.

At any rate, Thomas ended with:

"I am a believer in freezing development of the core protocols and building new functionality on top of them. I like NAT. I like Paul Francis. I think the IETF has been hijacked by the leftovers from the OSI standards committees. I don't know what you call that philosophy, besides "end2end originalist".

I like NAT.  I think this is Paul Francis.  The IETF has been hijacked by aliens, actually, and I'm getting a new tattoo:

On Flying Pigs, DNSSEC, and embedded versus overlaid security...

I found Thomas Ptacek's comments regarding DNSSEC deliciously ironic not for anything directly related to secure DNS, but rather a point he made in substantiating his position regarding DNSSEC while describing the intelligence (or lack thereof) of the network and application layers.

This may have just been oversight on his part, but it occurs to me that I've witnessed something on the order of a polar magnetic inversion of sorts.  Or not.  Maybe it's the coffee.  Ethiopian Yirgacheffe does that to me.

Specifically, Thomas and I have debated previously about this topic and my contention is that the network plumbing ought to be fast, reliable, resilient and dumb whilst elements such as security and applications should make up a service layer of intelligence running atop the pipes. 

Thomas' assertions focus on the manifest destiny that Cisco will rule the interconnected universe and that security, amongst other things, will -- and more importantly should -- become absorbed into and provided by the network switches and routers.

While Thomas' arguments below are admittedly regarding the "Internet" versus the "Intranet," I maintain that the issues are the same.  It seems that his statements below which appear to endorse the "...end-to-end argument in system design" regarding the "...fundamental design principle of the Intenet" are at odds with his previous aspersions regarding my belief.  Check out the bits in red.

Here's what Thomas said in "A Case Against DNSSSEC (A Matasano Miniseries):

...You know what? I don’t even agree in principle. DNSSEC is a bad thing, even if it does work.

How could that possibly be?

It violates a fundamental design principle of the Internet.

Nonsense. DNSSEC was designed and endorsed by several of the architects of the Internet. What principle would they be violating?

The end-to-end argument in system design. It says that you want to keep the Internet dumb and the applications smart. But DNSSEC does the opposite. It says, “Applications aren’t smart enough to provide security, and end-users pay the price. So we’re going to bake security into the infrastructure.”

I could have sworn that the bit in italics is exactly what Thomas used to say.  Beautiful.  If, Thomas truly agrees with this axiom and that indeed the Internet (the plumbing) is supposed to be dumb and applications (service layer) smart, then I suggest he should revisit his rants regarding how he believes the embedding security in the nework is a good idea since it invalidates the very "foundation" of the Internet.

I wonder what that'll do internal networks? 

That's all.  CSI is on.

/Hoff

(Written @ Home drinking Yirgacheffe watching UFC re-runs)