Huawei HQ in Shenzhen, China WASHINGTON: In a world where Chinese hackers steal everything from F-35 schematics to federal personnel files, why should we worry about Huawei? Because, cybersecurity experts explain, network routers, surveillance cameras and other widely sold devices from Huawei, Dahua, and other Chinese firms are riddled with vulnerabilities — flaws that are easy for attackers to exploit but hard for defenders to find, because they’re buried deep in what’s known as firmware. Traditional computer security techniques, already fallible enough with regular software, don’t work at all on firmware, which is loaded onto a device when it’s built, runs in the background largely hidden from the user, and can only be updated by the original manufacturer. Most devices networked together in the Internet of Things (IoT), in fact, have too little memory to run security scanning software or anything else besides their purpose-built firmware. But, Finite State founder Matt Wyckhouse and ReFirm Labs co-founder Terry Dunlap told me in interviews, there are now ways to run an automated search through firmware files to find suspicious code. What those automated watchdogs have found so far is disturbing. In a single 36-hour run, Finite State’s tool checked 1.5 million firmware files from 558 Huawei enterprise networking products — that’s just business systems, not consumer devices — and found the average device had 102 vulnerabilities, at least a quarter of them severe enough to let a hacker get full access easily. That’s much more than comparable Western products, Wyckhouse told me: “These are some of the worst devices we’ve ever tested.” It’s not just Huawei, Dunlap told me. In 2017, his ReFirm Labs team — some of them, including Dunlap himself, ex-NSA hackers — found a backdoor in the firmware of a surveillance camera made by Dahua, similar to one they’d discovered a few years before in a Huawei router. And the backdoor had been opened: Once ReFirm told their client (a Fortune 500 firm which they won’t name) what to look for, the company’s network operators discovered their Dahua cameras had been sending data out a rarely-used port, right through the company’s firewall, to unknown IP addresses in China. Dahua at first ignored ReFirm’s inquiries, then claimed the vulnerability was a simple error that had been fixed in the latest update. But when ReFirm looked through the updated firmware, they still found the same backdoor — just relocated in a different place in the code. (Huawei had done the same thing). “That just screams malicious to me,” Dunlap said, confirming that he was dealing with deliberate deception, not a mistake. Congress banned Dahua cameras from the US market the following year. So while Wyckhouse declines to say whether the vulnerabilities found by Finite State were the result of malice or incompetence, Dunlap has no hesitation: “We stand by the claim today that Dahua specifically has intentional back doors.” NSA headquarters The Art of the Hunt Terry Dunlap spent five years at the National Security Agency — which means the Chinese know everything about him. “With the OPM breach a few years ago by the Chinese,” which swept up extensive personnel files on intelligence personnel, “they have all my personal information, all my relatives’ personal information, everywhere I’ve lived within the past 10 years, all my financial information including my fingerprints,” he said. “So I’m totally exposed.” Working at NSA as a “state-sponsored hacker” (his own words) from 2002-2007, Dunlap said, “it was a very manual, intensive process using open source tools to look for these types of things in the operating systems or the firmware of these IoT devices. “With the previous company I owned, it took us like 10 years to automate this process,” Dunlap continued. Today at ReFirm, he said, “what normally takes weeks if not months to find, we’ve automated, where we can take the firmware in any of these IoT devices and in about 30 minutes get a complete profile on passwords that may have been accidentally left in, cryptographic keys that may or may not be warranted… insecure coding practices that could be exploited.” “It’s kind of like the Holy Grail to be able to automate that process,” said Finite State’s Wyckhouse, who worked on cyber contracts for defense and intelligence agencies at Battelle for 13 years. “What we do that’s novel is … vulnerability discovery on firmware at a very, very large scale” — like processing those 1.5 million Huawei files in 36 hours. Now, the easiest way to check for vulnerabilities is to analyze the source code, the original lines written by human programmers, which can be read and understood by another programmer. But while source code is available for open-source software — hence the name — companies guard their proprietary source code jealously, and Chinese companies are particularly unenthused about sharing their source code to Western cybersecurity experts. So what Finite State and ReFirm Labs have to work with is the object code — the machine-readable but human-mind-boggling string of 1s and 0s that you get after you compile the source code so it can actually run on a computer. What the two companies have created, each in its own way, are software tools that can extract the object code from the firmware of a device and hunt for suspicious patterns. At the most basic level, this means looking for things like credentials (say, a login:password combination) and cryptographic keys that can be used to authorize someone connecting to the device and getting access. Now, legitimate software engineers often put backdoors in code that they’re developing so they can easily access and debug it, and sometimes they just forget to take them out of the finished product; it’s similar to how game designers include cheat codes so they can, for example, debug Level 12 without having to beat Levels 1-11 first, and then enterprising players figure out how to activate these cheats in the published game. But the less clearly labeled a backdoor is, the better it is hidden, the more likely its presence is intentional rather than accidental. The next level of checking is to comb through the firmware for strings of code matching known vulnerabilities. That works both because legitimate but lazy programmers tend to copy the same flawed code over and over, especially if it’s open-source and therefore free, and because hackers likewise use the same exploits over and over. Ryan North explains Heartbleed The most difficult part: looking for new vulnerabilities no one’s ever seen before. Instead of matching specific lines of code known to be flawed, this requires actually seeing how the program would run and where that might go awry. For example, any function that sends data to the outside world, however legitimate the original purpose, can be subverted to send the wrong data to the wrong destination. Likewise, any code that allows a user or other outside source to enter data, for example — even just a password — is a potential point of vulnerability, since poorly written software can be tricked into accepting unauthorized commands instead of legitimate input. (Heartbleed, for example, worked by exploiting a sloppily configured input-output function, as explained by Ryan North’s webcomic xkcd in the image above). While an attacker has to work out how to exploit these loopholes with just the right input to get access, defenders just need to discover they exist. In the past, a human being would have to check for such weak points manually. Today, at ReFirm and Finite State, there is software that can automatically run a search — but a human still has to program that software so it knows what to look for. The next step, Dunlap said, is software that can program itself: machine learning algorithms that can be turned loose on code to teach themselves what vulnerabilities look like and how to find them. Such artificially intelligent security systems might even finally turn the tables in cyberspace and, for the first time, make defense more powerful than attack.
