January 31, 2008

Process Control Systems (SCADA and the like) & Virtualization

Securingscada Just when you get out, they pull you back in...

Jason Holcomb over at the Digital Bond blog posted something that attracted my attention.  He popped up an innocuous entry titled "Virtualization in the SCADA World: Part 1" which intrigued me for reasons that should be obvious to anyone who reads my steaming pile of blogginess with any sort of regularity.

It would be easy to knee-jerk and simply roll my eyes suggesting that adding virtualization to the "security by obscurity" approach we've seen being argued recently is just inviting disaster (literally,) but I'm trying to be rational about this.  I want to understand the other camp's position and learn from it, hopefully contributing to break down a wall or two...

Jason sets it up:

A few years back, the traditional IT world was debating the merits of virtualization. There were concerns about performance, security, vendor support, and a host of other issues. Fast-forward to today, however, and you’ll find virtual machines in use in nearly every data center.

I think it's fair to say that while most folks would be hard pressed to dispute the merits of virtualization, the concerns regarding "...performance, security, vendor support, and a host of other issues" are hardly resolved.  In fact, they are escalating.

<snip>

So what are the implications of this in the SCADA world? I think it’s just a matter of time before we see more widespread acceptance of VMware and other virtualization platforms in production control systems. The benefit here may be less about cost savings, though, and more about increased functionality. The ability to snapshot and clone machines for backup and testing, for example, is very attractive.

I think the paragraph above is extremely telling because it's really focused on debating the value proposition which is really a foregone conclusion for all the reasons Jason mentions.   The real meat will hopefully be discussed in the follow-on's:

We’re going to examine this subject over a series of blog posts. Hopefully we’ll cover all the major topics – security, reliability, performance, serial communication issues, vendor support, and adoption rate, to name a few.

I look forward to your comments and opinions.

In my first comment to Jason's posting, I alluded to a whole host of virtualization-related issues which are grounded in practice and not hype and asked that since SCADA security is billed as being SO much different than "IT security" what this intersection will bring and how one might assess risk (and against what.)

Further, given various C&A standards, I'm interested in how one might approach (depending upon industry) holding these systems up to a C&A process once virtualization is added to the mix. 

It will be an interesting discussion, methinks.

/Hoff

 

January 23, 2008

Pushing Reset On the IT vs. SCADA Security Debate....

Bigred I think that perhaps I have chosen a poor approach in trying to raise awareness for process control and SCADA (in)security.  You can find recent SCADA posts here, including the "awareness campaign" Mogull and I launched a couple of weeks back that got a ton of eyeballs ...

I believe I reacted poorly to the premise that some of those who assert expertise in this area tend to dismiss anyone who has a background only in what they define as "IT Security" as being unable to approach understanding -- let alone securing -- this technology.

Let me take a step back for a moment.

I'd like to get to the bottom of something regarding the alleged great divide between what is being described as diametrically opposed aptitude and experience required to secure "IT" infrastructure versus process control systems such as SCADA. 

I notice a similar divergence and statements being made between those who specialize in web application security (WebAppSec) versus information or network security (InfoSec/NetSec.)

For example, WebAppSec is a discipline and specialty that some suggest requires a level of experience and expertise that goes beyond that of traditional "information security" or "network security" practitioners.  It is suggested that in order to truly secure web applications, one generally requires programming experience, understanding complex data structures, databases, distributed application architecture, etc., at a very detailed level.

I think these statements are reasonable, but does it preclude an InfoSec/NetSec practitioner from contributing to effectively manage risk in a WebAppSec environment?

A network security practitioner can deploy a web application firewall and generally configure the solution, but the antagonists suggest that in order to provide a level of protection commensurate with the complexity and dynamics of the code which they are attempting to "secure," it cannot be done without an in-depth understanding of the application, it's workflow and behavior.

Again, in reflection, I'd say that's not an unreasonable assessment.  However, WebAppSec and NetSec/InfoSec guys in mature organizations generally should know what they don't know and work together to implement a holistic solution across layers.  It doesn't always work out that way, but in order to secure the WebApps, we can't ignore the underpinnings of the network or information security foundations, either. 

It really should be a discussion, then, on how to unify complimentary approaches at various levels with an overall focus on managing risk. However, what I find is a downright civil war on the "IT" vs "SCADA" security front.  I have to ask why?

Here's an excerpt from a post I found on Dale Peterson's excellent Digital Bond blog.  It was a review of a SCADA security presentation at a CCC event in Italy regarding an introduction (of sorts) to SCADA security.  The premise isn't really important, but I think that this does a good job of explaining some of the issues and sentiments that I am referring to:

Now here’s the good news: Asset owners, you don’t need to worry about hackers. When they talk about “owning critical infrastructure”, they’re just sharing their wildest dreams. In reality, they have nothing in their hands. Zero. Nada. Niente. It will take several more years until the hacker community has learned to master various flavours of PLCs with their different protocols and vulnerabilities. It will take further years until they get to things like OPC and furnish advanced attack methods against it. And by the time they come up with decent exploits for the various SCADA applications that we use today, most CxOs will already be retired. We have heard over and over again that the IT folks aren’t particularly good at securing SCADA environments. Guess what, they aren’t good at attacking them either. However our hackers do think nobody will notice because the stuff is all so complex. That’s what I call “insecurity by obscurity”.

This whole notion of "it's so complex and so few people know anything about it so we have nothing to fear" seems to be the point of divide.

There's another really telling post on Dale's site (authored by him) titled "Firewalls are easy, control systems are hard" wherein the following inaccurate premise is painted which reduces the scope of the entire infosec/netsec profession down to a five-tupule in a basic packet filtering firewall:

One of the common refrains heard again at ISA Expo is that IT firewalls are too difficult to configure and deploy. Several presenters, especially those promoting field security appliances, mentioned this, and it seemed to be generally accepted. While I’m all for simplicity and credit the vendors for trying to ease deployment, firewalls are simple compared to the deploying PLC’s, defining points in the SCADA database, developing displays, control loops, and the myriad of other detailed configuration required to make a control system work.

A firewall ruleset is as simple as defining rules by source IP, destination IP and port. Since communication in control systems is limited as compared to the corporate network, the ruleset is usually very small.

How simple is that compared to monitoring and controlling a complex process distributed over a plant or large part of the country with 5000 points or 100,000 points? I was introduced to control systems in 2000 and have worked on a large number of SCADA and DCS in a variety of industry sectors and I still marvel at the effectiveness and attention to detail in these systems. There is nothing in firewall or any other IT security system configuration that comes close to the complexity in configuring and deploying control systems

That maybe true in an SME/SMB network, but the reality is now that firewalls in a large enterprise (which is a much more reasonable comparison) are just a small piece of the puzzle.  Endpoints numbering in the thousands (if not tens of thousands) which run hundreds of application combinations aren't exactly chopped liver to secure.  Add in Web Application Firewalls, Database Monitoring, Encryption (at rest, in motion,) IDS, IPS, Proxies, A/V, URL Filtering, Anti-spam, NBAD, SIEM, etc. and it just gets more complex from there.

If life were as simple as deploying a firewall and firing off a five-tuple ruleset, we wouldn't be in this pickle.

Regardless of whether a NetSec/InfoSec practitioner knows in-depth details regarding implementing PLC's/RTU's or the inner-workings of the IEC61131-3 block programming language is neither here nor there because it's only one piece of the puzzle. 

Many InfoSec/NetSec practitioners don't have expertise in SQl/pSQL, but they work with the DBA's to secure databases, right?

Once these systems are interconnected to an IP-enabled network, it requires cooperation up and down the stack.  InfoSec/NetSec pro's need to become SCADA-aware and SCADA pro's need to stop suggesting that this technology is just so complex and overwhelming that it's beyond our ability to effectively collaborate and that "firewall jockeys" just can't understand.

The reality is that the bad guys look for the weakest link in the chain.  Will they attack complex protocol stacks and programming languages first?  No, they'll go after the low-hanging fruit like poorly-configured/secured end-nodes, bad perimeter controls and general user-driven crap like we see in the rest of the world.  They won't need to even spell PLC.

We need the same level of information sharing and respective skill set cross-pollinization in this regard instead of squaring off like it's a battle between us versus them. 

/Hoff

January 18, 2008

CIA: Hackers to Blame for Power Outages ('nuff said)

Aurora I'm sorry, did someone say we have nothing to worry about when it comes to SCADA and control systems security?  I must have missed the memo:

CIA: Hackers to Blame for Power Outages

WASHINGTON (AP) — Hackers literally turned out the lights in multiple cities after breaking into electrical utilities and demanding extortion payments before disrupting the power, a senior CIA analyst told utility engineers at a trade conference.

All the break-ins occurred outside the United States, said senior CIA analyst Tom Donahue. The U.S. government believes some of the hackers had inside knowledge to cause the outages. Donahue did not specify what countries were affected, when the outages occurred or how long the outages lasted. He said they happened in "several regions outside the United States."

"In at least one case, the disruption caused a power outage affecting multiple cities," Donahue said in a statement. "We do not know who executed these attacks or why, but all involved intrusions through the Internet."

A CIA spokesman Friday declined to provide additional details.

"The information that could be shared in a public setting was shared," said spokesman George Little. "These comments were simply designed to highlight to the audience the challenges posed by potential cyber intrusions."

Donahue spoke earlier this week at the Process Control Security Summit in New Orleans, a gathering of engineers and security managers for energy and water utilities.

The Bush administration is increasingly worried about the little-understood risks from hackers to the specialized electronic equipment that operates power, water and chemical plants.

In a test last year, the Homeland Security Department produced a video showing commands quietly triggered by simulated hackers having such a violent reaction that an enormous generator shudders as it flies apart and belches black-and-white smoke.

The recorded demonstration, called the "Aurora Generator Test," was conducted in March by government researchers investigating a dangerous vulnerability in computers at U.S. utility companies known as supervisory control and data acquisition systems. The programming flaw was fixed, and equipment makers urged utilities to take protective measures.

Now, this article says these attacks were outside the U.S. (since it came from the CIA, you can imagine why.)  Also, it does NOT directly say that SCADA systems were attacked.  However, these statements were made at a SCADA "Process Control" Security conference, so I'm going to take the liberty of bridging that assumption.  Either way, it highlights the problem at hand (see the 787 Dreamliner story and the Polish Tram derailment...)

Do y ou really think it's that much of a reach to suggest it's not happening on our shores?

If anyone gives me any more crap about being concerned regarding the possibility/potential for disruption...look at the boldfaced section.  The compromise was conducted over the Internet.  Don't forget, this sort of thing is supposed to be impossible given some comments from my "awareness campaign":

Oh gosh, where do I begin Chris? 

What do the first letters of SCADA stand for?  Supervisory Control. 

A real SCADA system doesn't issue direct controls. It issues Supervisory Controls. There should be no time critical control loops in SCADA. In other words, we have vulnerabilities. But they won't destroy anything right away. We engineers know better than to trust complex software.

Most good design practice is based upon graceful degradation. In other words, we don't send a command to open a valve. We send commands to change the pressure differential setpoint. A local controller takes care of the rest. There are sanity checks in the local controller.

You could send commands to the field that would screw things up. But most people would notice and we'd take action. Keep in mind, that while our operation is very careful and deliberate, the distribution system was built for some wild extremes including pipe breaks, extreme weather, communcation outages, and vandalism. A successful attack would require intimate knowledge of where the real vulnerabilities are.

Are you an expert at water utilities too? 

No, Jake.  I'm not a water utilities expert, just a concerned observer & citizen. 

Hat tip to Stiennon for the source.

/Hoff

January 11, 2008

How To Say "Whoops! We Need To Rethink Our Train System's Control Plane" In Polish...

TrainwreckMy dear friend Murray (sorry if that expression of warmth comes as a surprise, Murr...) sent me this story from the Register:

Polish teen derails tram after hacking train network

A Polish teenager allegedly turned the tram system in the city of Lodz into his own personal train set, triggering chaos and derailing four vehicles in the process. Twelve people were injured in one of the incidents.

The 14-year-old modified a TV remote control so that it could be used to change track points, The Telegraph reports. Local police said the youngster trespassed in tram depots to gather information needed to build the device. The teenager told police that he modified track setting for a prank.

"He studied the trams and the tracks for a long time and then built a device that looked like a TV remote control and used it to manoeuvre the trams and the tracks," said Miroslaw Micor, a spokesman for Lodz police.

"He had converted the television control into a device capable of controlling all the junctions on the line and wrote in the pages of a school exercise book where the best junctions were to move trams around and what signals to change.

"He treated it like any other schoolboy might a giant train set, but it was lucky nobody was killed. Four trams were derailed, and others had to make emergency stops that left passengers hurt. He clearly did not think about the consequences of his actions," Micor added.

Transport command and control systems are commonly designed by engineers with little exposure or knowledge about security using commodity electronics and a little native wit. The apparent ease with which Lodz's tram network was hacked, even by these low standards, is still a bit of an eye opener.

Problems with the signalling system on Lodz's tram network became apparent on Tuesday when a driver attempting to steer his vehicle to the right was involuntarily taken to the left. As a result the rear wagon of the train jumped the rails and collided with another passing tram. Transport staff immediately suspected outside interference.

The youth, described by his teachers as an electronics buff and exemplary student, faces charges at a special juvenile court of endangering public safety. ®

Yes, yes.  I know, it's not a SCADA system...as fun as that would be to bring up again, I don't need any death threats, so I won't mention it...directly.  But if you read about the recent security design debacle of the Boeing 787 Dreamliner and then look at this, it doesn't take much of a logic jump to see why we should be worried about how command/control systems are implemented.

My next piece of chicanery is to steal one of Mogull's Wii Guitar Hero controllers, hack it, and cause it to electrocute his cat every time he hits C# on Stairway to Heaven...

/Hoff

December 13, 2007

Breaking News: Successful SCADA Attack Confirmed - Mogull Is pwned!

Scada A couple of weeks ago, right after I wrote my two sets of 2008 (in)security predictions (here and here), Mogull informed me that he was penning an article for Dark Reading on how security predictions are useless.  He even sent me a rough draft to rub it in.

His Dark Reading article is titled "The Perils of Predictions - and Predicting Peril" which you can read here.  The part I liked best was, of course, the multiple mentions that some idiot was going to predict an attack on SCADA infrastructure:

Oh, and there is one specific prediction I'll make for next year: Someone will predict a successful SCADA attack, and it won't happen. Until it does.

So, I'm obviously guilty as charged.  Yup, I predicted it.  Yup, I think it will happen.

In fact, it already has...

You see, Mogull is a huge geek and has invested large sums of money in his new home and outfitted it with a complete home automation system.  In reality, this home automation system is basically just a scaled down version of a SCADA system (Supervisory Control and Data Acquisition.)  Controlling sensors and integrating telemetry with centralized reporting and control...

Rich and I are always IM'ing and emailing one another, so a few days ago before Rich left town for an international junket, I sent him a little email asking him to review something I was working on.  The email contained a link to my "trusted" website.

The page I sent him to was actually trojaned with the 0day POC code for the QT RTSP vulnerability from a couple of weeks ago.  I guess Rich's Leopard ipfw rules need to be modified because right after he opened it, the trojan executed and then phoned home (to me) and I was able to open a remote shell on TCP/554 right to his Mac which incidentally controls his home automation system.  I totally pwn his house.

CctvSo a couple of days ago, Rich went out of town and I waited patiently for the DR article to post.  Now that it's up, I have exacted my revenge.

I must say that I think Rich's choice of automation controllers was top-shelf, but I think I might have gone with a better hot tub controller because I seem to have confused it and now it will only heat to 73 degrees.

I also think he should have gone with better carpet.

I'm pretty sure his wife is going absolutely bonkers given the fact that the lights in the den keep blinking to the beat of a Lionel Ritchie song and the garage door opener keeps trying to attack the gardener.  I will let you know that I'm being a gentleman and not peeking at the CCTV images...much.

Let this be a lesson to you all.  When it comes to predicting SCADA attacks, don't hassle the Hoff!

/Hoff

December 03, 2007

2008 Security Predictions -- They're Like Elbows...

Carnacangled_2 Yup.  Security predictions are like elbows.  Most everyone's got at least two, they're usually ignored unless rubbed the wrong way but when used appropriately, can be devastating in a cage match...

So, in the spirit of, well, keeping up with the Jones', I happily present you with Hoff's 2008 Information (in)Security Predictions.  Most of them are feature attacks/attack vectors.  A couple are ooh-aah trends.  Most of them are sadly predictable.  I've tried to be more specific than "cybercrime will increase."

I'm really loathe do these, but being a futurist, the only comfort I can take is that nobody can tell me that I'm wrong today ;)

...and in the words of Carnac the Magnificent, "May the winds of the Sahara blow a desert scorpion up your turban..."

  1. Nasty Virtualization Hypervisor Compromise
    As the Hypervisor gets thinner, more of the guts will need to be exposed via API or shed to management and functionality-extending toolsets, expanding the attack surface with new vulnerabilities.  To wit, a Hypervisor-compromising malware will make it's first in-the-wild appearance to not only produce an exploit, but obfuscate itself thanks to the magic of virtualization in the underlying chipsets.  Hang on to yer britches, the security vendor product marketing SpecOps Generals are going to scramble the fighters with a shock and awe campaign of epic "I told you so" & "AV isn't dead, it's just virtualized" proportions...Security "strategery" at it's finest.

  2. Major Privacy Breach of a Social Networking Site
    With the broadening reach of application extensibility and Web2.0 functionality, we'll see a major privacy breach via social network sites such as MySpace, LinkedIn or Facebook via the usual suspects (CSRF, XSS, etc.) and via host-based Malware that 0wns unsuspecting Millenials and utilizes the interconnectivity offered to turn these services into a "social botnet" platform with a wrath the likes of which only the ungoldly lovechild of Storm, Melissa, and Slammer could bring...

  3. Integrity Hack of a Major SaaS Vendor
    Expect a serious bit of sliminess to occur with real financial impact to occur from a SaaS vendor's offering.  With professional cybercrime on the rise, the criminals will go not only where the money is, but also after the data that describes where that money is.  Since much of the security of the SaaS model counts on the integrity and not just the availability of the hosted service, a targeted attack which holds hostage the (non-portable) data and threatens its integrity could have devastating effects on the companies who rely on it.  SalesForce, anyone?
     
  4. Targeted eBanking Compromise with substantial financial losses
    Get ready for a nasty eBanking focused compromise that starts to unravel the consumer confidence in this convenient utility; not directly because of identity abuse (note I didn't say identity theft) but because of the business model impact it will bring to the banks.   These types of direct attacks (beyond phishing) will start to push the limits of acceptable loss for the financial institutions and their insurers and will start to move the accountability/responsibility more heavily down to the eBanker.  A tiered service level will surface with greater functionality/higher transaction limits being offered with a trade-off of higher security/less convenience.  Same goes for credit/debit cards...priceless!

  5. A Major state-sponsored espionage and cyberAttack w/disruption of U.S. government function
    We saw some of the more noisy examples of low-level crack attacks via our Chinese friends recently, but given the proliferation of botnets, the inexcusably poor levels of security in government systems and network security, we'll see a targeted attack against something significant.  It'll be big.  It'll be public.  It'll bring new legislation...Isn't there some little election happening soon?  This brings us to...

  6. Be-Afraid-A of a SCADA compromise...the lunatics are running the asylum!
    Remember that leaked DHS "turn your generator into a roman candle" video that circulated a couple of months ago?  Get ready to see the real thing on prime time news at 11.  We've got decades of legacy controls just waiting for the wrong guy to flip the right switch.  We just saw an "insider" of a major water utility do naughty things, imagine if someone really motivated popped some goofy pills and started playing Tetris with the power grid...imagine what all those little SCADA doodads are hooked to...
     
  7. A Major Global Service/Logistics/Transportation/Shipping/Supply-Chain Company will be compromised via targeted attack
    A service we take for granted like UPS, FedEx, or DHL will have their core supply chain/logistics systems interrupted causing the fragile underbelly of our global economic interconnectedness to show itself, warts and all.  Prepare for huge chargebacks on next day delivery when all those mofo's don't get their self-propelled, remote-controlled flying UFO's delivered from Amazon.com.

  8. Mobile network attacks targeting mobile broadband
    So, you don't use WiFi because it's insecure, eh?  Instead, you fire up that Verizon EVDO card plugged into your laptop or tether to your mobile phone instead because it's "secure."  Well, that's going to be a problem next year.  Expect to see compromise of the RF you hold so dear as we all scramble to find that next piece of spectrum that has yet to be 0wn3d...Google's 700Mhz spectrum, you say? Oh, wait...WiMax will save us all...
     
  9. My .txt file just 0wn3d me!  Is nothing sacred!?  Common file formats and protocols to cause continued unnatural acts
    PDF's, Quicktime, .PPT, .DOC, .XLS.  If you can't trust the sanctity of the file formats and protocols from Adobe, Apple and Microsoft, who can you trust!?  Expect to see more and more abuse of generic underlying software plumbing providing the conduit for exploit.  Vulnerabilities that aren't fixed properly combined with a dependence on OS security functionality that's only half baked is going to mean that the "Burros Gone Wild" video you're watching on YouTube is going to make you itchy in more ways than one...

  10. Converged SensorNets
    In places like the UK, we've seen the massive deployment of CCTV monitoring of the populous.  In places like South Central L.A., we have ballistic geo-location and detection systems to track gunshots.  We've got GPS in phones.  In airports we have sniffers, RFID passport processing, biometrics and "Total Recall" nudie scanners.  The PoPo have license plate recognition.  Vegas has facial recognition systems.  Our borders have motion, heat and remote sensing pods.  start knitting this all together and you have massive SensorNets -- all networked -- and able to track you to military precision.  Pair that with GoogleMaps/Streets and I'll be able to tell what color underwear you had on at the Checkout counter of your local Qwik-E-Mart when you bought that mocha slurpaccino last Tuesday...please don't ask me how I know.

  11. Information Centric Security Phase One
    It should come as no surprise that focusing our efforts on the host and the network has led to the spectacular septic tank of security we have today.  We need to focus on content in context and set policies across platform and transport to dictate who, how, when, where, and why the creation, modification, consumption and destruction of data should occur.  In this first generation of DLP/CMF solutions (which are being integrated into the larger base of "Information" centric "assurance" solutions,) we've taken the first step along this journey.  What we'll begin to see in 2008 is the information equivalent of the Mission Impossible self-destructing recording...only with a little more intelligence and less smoke.  Here come the DRM haters...
     
  12. The Attempted Coup to Return to Centralized Computing with the Paradox of Distributed Data
    Despite the fact that data is being distributed to the far reaches of the Universe, the wonders of economics combined with the utility of some well-timed technology is seeing IT & Security (encouraged by the bean counters) attempting to reel the genie back in the bottle and re-centralize the computing (desktop, server, application and storage) experience back into big boxes tucked safely away in some data center somewhere.  Funny thing is, with utility/grid computing and SaaS, the data center is but an abstraction, too.  Virtualization companies will become our dark overlords as they will control the very fabric of our digital lives...2008 is when we'll really start to use the web as the platform for the delivery of all applications, served through streamed desktops on thinner and thinner clients.

So, that's all I could come up with.  I don't really have a formulaic empirical model like Stiennon.  I just have a Guiness and start complaining.  This is what I came up with.

In more ways than one, I hope I'm terribly wrong on most of these.

/Hoff

[Edit: Please see my follow-on post titled "And Now Some Useful 2008 Information Survivability Predictions" which speak to some interesting less gloomy things I predict to happen in 2008]

My Photo

Lijit Search

Disclaimer

  • The views and opinions expressed here are those of Christofer Hoff only and in no way represent the views, positions or opinions - expressed or implied - of my employer or anyone else.

July 2008

Sun Mon Tue Wed Thu Fri Sat
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    

Categories