In 2015, Russia hacked the Ukraine power grid. Are we next? (Part 1 of 3)

In a widely publicized episode, the electric power grid in Ukraine was attacked during late 2015 and 2016, using computer viruses that destabilized the supervisory control and data acquisition  (SCADA) system.  Experts are about as certain as one can be that the Russian government was behind the episode.

Could this occur in the United States, and if so, what could we do about it?  If it happens, should we assume that the attack originated in Russia, like the Ukrainian event?

To answer such a question, we really need to break it down.  Here, I'll start with the basics: what do we really know about the Ukraine attack?  My sources, though, are unclassified (in fact there is one Wired article that had such detail that I'm tending to trust it heavily).  I mention this because we do live in an era of fake news, and there could be a deeper layer of insights, not be available to me.  For a question of national aggression -- literally an act of war by one country against another -- one needs to go deep and not trust the superficial!

But I'm just a regular guy without a clearance, and if I knocked on doors at the CIA and NSA, I wouldn't get very far.  Here's what I've learned from public materials.

First, in the wake of the event there were a series of very reputable groups that flew to Ukraine and participated in really careful studies of the precise modality of the attack.  The can be little doubt that the attack was extremely sophisticated and carefully planned, that Ukraine was not some sort of banana republic with an incompetent management of its national grid (it turns out that Ukraine was highly professional and pretty close to the state of the art).  From this, we'll have little choice but to acknowledge that the US grid is probably vulnerable, too.

Ukraine built its grid during a period when it was fairly wealthy, and was in a position to buy cutting edge technologies.  The system was a relatively standard high-quality SCADA solution, obtained from the same vendors who sell such systems here in the US.  Moreover, the country knew of the threat from outside, and managed its system quite professionally, using a "military" security standard.   However, Ukraine wasn't the most paranoid you could imagine.  In particular, it did allow operators to use computers attached to its network to receive email with links and attachments, and they could access Internet web sites from their office computers. 

Apparently, this was the first portal the attackers leveraged: they sent some form of normal looking work-related email, but it lured operators to a poisoned web site, which downloaded a virus that connected back to the attacker control system.  The trick being: the web site did whatever it was nominally supposed to do, so the operators never realized they had been compromised.

For example, think of the first time you visited the real-time market feed data site provided by your bank or retirement fund: you probably agreed to install some form of web browser plug-in to see the animated graphs of market activity.  The first step of the Ukraine attack was a bit like that, but that plug-in (in addition to doing what it promised), did a bit more.

Before assuming that this first step already rules out such an attack in the US, and that this could never happen here, one has to pause and realize that many of us receive emails from the HR organizations of our employers that require clicking links.   Many of us work for companies that use plug-ins to offer all sorts of functionality through browser extensions.  In fact, I work for such a company: Cornell University does this too.  If you were familiar with Cornell's web page layouts and logos, and knew how to compose a professional looking email with the right content, even a security-conscious person like me might follow the link without much thought. 

The core insight is that because of the so-so state of security on our computer operating systems, web browsers and other technologies, even normal news sites and other mundane web sites can potentially be a launch-point for attacks.   So this first step of the Ukraine attack could be successful in the US too, today.  In fact I know of very similar events that led to intrusions right into top-secret DoD systems and ones used within the White House, and that's without even having access to the classified version of the picture.  This definitely could happen in the US, even in highly sensitive systems.

Ok, but in fact, Ukraine's office computers weren't actually connected to the SCADA systems.  In fact the vast majority of  the state-operated power grid company employees had mundane jobs, like taking new orders, billing, scheduling repair crews.  Breaking into their computers wouldn't lead anywhere. So what happened next?

The initial exploit gave the attackers a toehold: it left them with hooks they could use to (in effect) log into a few computers, inside the Ukraine power grid operations center, but not ones concerned with actual power grid operations.  Those systems were much better protected.  So, our hackers needed to break through a second firewall.

As I understand it, this required finding systems used by operators who actually had permissions to log into the SCADA network.  Apparently, the Ukraine system masked the roles of the computers, and figuring this out wasn't simple and took months.  Nonetheless, step by step, the intruders managed to identify several such systems. 

In their next step, it seems that the hackers used so-called root kits to attack these computers from inside the Ukraine power system corporate network.  A root kit is a package of software, collected by hackers over decades, that takes advantage of subtle software bugs to sneak into a computer and grant the attacker superuser control, unnoticed by the owner of the machine.  There are a surprisingly large number of such systems -- you can download dozens from the web.  And then beyond that are specialized ones created by national intelligence services: they often use vulnerabilities that their designers discovered, and that nobody else was even aware of.

Software has bugs, and older software systems are worst of all.  Don't fool yourself: any system you can buy or use today has vulnerabilities.  Some are child's play to break into, and some are much more resilient, but none is foolproof.

So, our intruders laid low, figured out which machines to attack, and then finally after months of effort, managed to compromise a machine with VPN access to the SCADA platform.  But VPN software expects passwords and often more: two-factor systems that use RDA keychain dongles, fingerprints, special cards -- all such things are common.  I take it that Ukraine was using such a system.

To circumvent those issues, the trick is to modify the operating system itself, so that the next time a legitimate operator logs into the VPN, you can ride along with him or her, sneaking in for a little while and then ultimately, if luck is on your side, to leave a subtle open doorway, perhaps in the form of a legitimate-looking data feed that actually is carrying your covert traffic. 

So, after waiting for someone to activate the VPN from that machine, our attackers eventually managed to leapfrog into the secured environment, at some instant when the VPN connection was open.  Moreover, once in, even more steps were needed to actually penetrate and ultimately, incapacitate the SCADA platform, and those had to occur without site security systems noticing the intrusion.   They apparently had further layers of passwords to crack (here, access to a supercomputer can be helpful), and all of this had to occur without tripping the monitoring systems.

So who was behind this?  President Trump talks often about the 300lb pimply kid sitting in his bedroom.  Was it him?

Notice that the first steps required fluency in written Ukranian, and detailed knowledge of HR emails and other corporate emails within the organization.  Subsequent steps required knowing the software systems and versions and their vulnerabilities, and there may even have been a step at which a supercomputer was used to break a password by brute force.  Step by step, one would have to understand what monitoring tools were in use and how to avoid detection by it.  Military-quality root kits aren't so easy to come by.

The bottom line?  It is unquestionable that Russia was behind this attack.   And Russia happened to benefit from it too, enormously.  They had the means, the motive, and the timing coincided with a major flare-up in military tensions between Russia and Ukraine over Crimea, which Russia had just annexed.

The imaginary fat kid could never have managed this.  What we see here is how very sophisticated exploit that took years to prepare was carried out very slowly and deliberately.  Without getting even more detailed, investigators were able to show that the intruders had debugged the exploit over an extended period, using all sorts of pre-exploit testing and trial runs of aspects of the ultimate attack.  Even attack software needs to be debugged!

Ukraine was victim of a genuinely sophisticated team of expert attackers, backed by a country or organization with massive national resources, and the patience to chip away at the system for literally years, entirely undetected.  And that country was Russia.

The US media has tended to portray Ukraine as a backwoods country that set itself up for trouble, but as I read the story, that interpretation is totally invalid.  Most of what I've described definitely could occur in the US.   Our power grid operators may have plugged the specific holes that have now been identified in the products deployed in Ukraine, and may be better at monitoring the system and applying patches, but honestly, I wouldn't bet too heavily on that sort of thing. 

I once was invited to an unclassified NSA briefing in which the question of breaking into systems came up.  Their expert said that well, he couldn't get specific, but that honestly, it was impossible to build a system today that his people couldn't break into.  He said that modern computers have millions of lines of code, not to mention devices of many kinds that themselves include computers (routers, printers, even network interface cards).  NSA had made a science out of finding back doors in. 

He said that we should imagine a little village where the houses all had bowls of jewels on the main dining room table.  And all the doors and windows are wide open.  Even if they weren't, most of the windows have no latches and the doors have locks that share a single key.  And even if you fixed the doors and windows, the walls themselves are made of plywood screwed into wood beams, and with a screwdriver and a few minutes, you could make a new door just for yourself.  To say nothing of using a ladder to try breaking in upstairs, where the air conditioners turn out to not actually be attached and can just be pushed out of their slots.

What the US NSA can do, the Russian intelligence service can do as well.  Plus, they have tons of people who are fluent in Ukrainian human resources memo-writing.

So could the same thing happen to us?  Sure, without question.

Oddly, our main strength isn't that we operate our systems better or that we monitor them better.  We don't, and it isn't for lack of trying: these kinds of systems can't be defended against that sort of attack.

Our real advantage (a small one) is that to compromise our entire national grid, all at once, you would need to pull off at least 10 and perhaps more like 25 Ukraine-style attacks, because there are roughly 10 large scale regional transmission operators and independent system operators (RTOs and ISOs), and they work with an additional smaller 15 or so transmission operating entities.  Each makes its own technical choices, although there are some popular technologies that are near monopolies in their particular roles.  Thus what worked in Ukraine probably could work here, but might "only" knock out some subset of the overall national grid.

 (But this is plenty for one blog entry, so lets pause for a quick coffee and then we can resume...)