WASHINGTON: Two years ago, Capt. John Zimmerman and his award-winning Navy team were testing a software upgrade for submarines when they ran into a surprising problem. When they changed the code controlling the Tomahawk missile launchers, the torpedo tubes stopped working.

Fortunately, this all happened in a laboratory ashore before the upgrade got anywhere near the fleet. No sub ever experienced the torpedo malfunction. (This is why you test upgrades in the lab in the first place).

But the experience showed how hard traditional military systems can be to update. Every element is tightly integrated with everything else, like the old song about “the toe bone’s connected to the foot bone, the foot bone’s connected to the ankle bone….” If one component becomes obsolete, you can’t just pull it out and swap in something new, because you might break something apparently unrelated, as with the missile launchers and torpedo tubes on Zimmerman’s simulated sub.

To make upgrades easier, cheaper, and, most importantly, faster — so we can stay ahead of rapidly advancing adversaries like Russia and China — the Defense Department has embraced an approach called “open architecture.” Open architectures aren’t exactly new, but the military’s enthusiasm for them is. The ideal is a modular system, where any component from any company can plug-and-play as long as it complies with certain clearly defined standards.

That’s what the Navy’s trying now with subs, for example. “The submarine force recognized that operating with a legacy weapon control system didn’t make sense,” Zimmerman told me. “It costs too much to make changes and bring new capabilities with the legacy architecture, so that system is being re-architected to make it a modern open-architecture system.”

Of course, moving to open architecture is much, much easier said than done. At a couple of recent conferences — one on open architecture specifically, the other on military robotics — I was struck by the consensus among defense officials that open architecture is (1) very important but (2) very, very hard.

“We all seem to be moving a hundred miles a minute on a terrific-sounding idea,” said Kristen Baldwin, principal deputy to the deputy assistant secretary of Defense for systems engineering. “What we found out immediately was one of the biggest problems is you can’t just simply say ‘be modular and open’ or ‘implement this on standard’ because the context matters and the details of the implementation matter.”

Kristen Baldwin

Consider a bomb-squad robot called AEODRS, for Advanced Explosives Ordnance Disposal Robotic Systems. “It was designed from the ground up to be an open architecture modular system,” said Thomas Dee, the deputy assistant secretary of the Navy who oversees the program. “It’s harder than it sounds,” he said. “Everybody talks about open architectures. Actually doing it proved to be a little more difficult than we expected, quite honestly.”

One problem was making sure the government’s interface standards were well enough defined — and well enough understood — by the industry participants so that their components really would work together. Otherwise there’d be a tremendous mess in the lap of the company acting as lead systems integrator, Northrop Grumman, which had to put everything together.

“We didn’t really appreciate the risk that we were asking that systems integrator to take on,” Dee said. “I think we’ve got that now.” After the Navy withdrew the initial Request For Proposals, rewrote the RFP, and reissued it, a first-stage AEODRS contract was awarded August 31.

The National Security Agency had a similar problem, said James Thompson, the NSA’s chief of “global access operations.” (“Access” is NSA Newspeak for “spy on.”) “I can’t really go into specifics of the actual project,” he said, “but we had a situation where we started with our software defined radio framework, and just sort of released it into the wild and said, everybody bring your ideas.”

The result? A hundred flowers bloomed — but the garden was unmanageable. “We reduced our development time for the initial capability, but the integration time [to make everything work together] went up exponentially,” Thompson said: The supposedly high-speed project took 18 months. Ultimately, much like the AEODRS program, NSA tried again with a much “better defined” architecture, and this second attempt took just two months.

The same issues replicate at the largest scale with the Defense Department’s massive consolidation of its numerous networks into a single Joint Information Environment, said Maj. Gen. Sarah Zabel, vice-director of the Defense Information Systems Agency (DISA). As military services and agencies merge their networks into the JIE, they’re supposed to either adopt JIE-provided services or build their own “in accordance with the [JIE] open architecture, but it’s hard, really hard in the details,” she said. Notably, the 11 Defense Enterprise Computing Centers (DECCs) all meet the same standards, but they’re still not fully compatible with each other.

“The reality on the ground is we have 11 different DECCs and they’re each implemented in a different manner,” Zabel said. “They correspond to the architecture….and yet they’re all different, they’re all unique, and we find interoperability challenges between them.”

So open architecture is great, but there’s also such a thing as too open. “It can be so open that there’s too many options and you’re incompatible,” said Col. Linda Jantzen, the Army’s Chief Data Officer. On the other hand, she added, “you can be not open enough, so that nobody knows what you want or what you can build to.”

A particularly acute problem Baldwin and others highlighted: data rights. Traditionally, a defense contractor will build a tightly integrated system for which it owns all the data, guaranteeing itself a monopoly on future upgrade work because it’s the only competitor that understands the system well enough to work on it.

This model offers a decades-long revenue stream. It’s also completely incompatible with open architecture. But when government agencies tried to take the diametrically opposite approach by taking ownership of the data and making it available to everyone, that effectively asked companies to share their secret formulas with competitors.

“The problem…was blind implementation of ‘I want to have all the data rights,’ not being thoughtful about what data is important to own the rights for,” said Baldwin.

“I see it as the difference between the what and the how,” said Zabel. The government needs to own the standards and specifications for what the system must do — but the contractor can own the trade secrets of how they achieve those goals. “That’s a simple answer to what’s actually a very complex problem,” she admitted.

“With open architecture you do not have to sell and give away your intellectual property,” said Brian Skibba of the Air Force Civil Engineer Center. “You actually can compartmentalize your intellectual property inside of a subroutine where nobody can see it,” he said, as long as that subroutine plugs smoothly into the open architecture. “I don’t have to ever see your code, I don’t have to see your algorithm, that’s yours,” he said. “It just has to be able to work.”

Call it an Oreo approach. The contractor’s proprietary information is the sweet center, surrounded and concealed by interfaces compatible with the open architecture.

Even with those Oreos, though, striking a balance between companies’ data rights and the government’s is still not easy. All told, getting open architecture to work requires getting both industry and government to adopt new mindsets, new business models, and new kinds of contract. It’s hard. But is it worth it?

Just consider the Air Force’s Open Mission Systems (OMS) demonstration on the B-2 bomber, U-2 spyplane, and Global Hawk drone, among others, said Baldwin. OMS has made it possible to “add software and payloads to these platforms… within weeks as opposed to months or years.”

Such tightened timelines are particularly important for cybersecurity, where new threats emerge at lightning speed. But once again, mindlessly chanting “open architecture” just won’t work. In some ways, open architecture is an easier target for adversaries, because it often includes commercial components that hackers can easily obtain and reverse-engineer in search of weaknesses, said Vern Boyle, director of technology in the cyber division of Northrop Grumman Information Systems. Today, he said of open systems, “they are more vulnerable — but they don’t have to stay that way.”

In the future, Boyle said, “can open architectures enable a rapid response to an evolving threat? I think, absolutely, yes, they can, but…it requires a fundamental shift in our protection strategy.”

Currently, computer networks tend to be static and cyber defenses reactive, only changing when a new threat is detected — by which time it may have already done its damage. By contrast, modern malware is increasingly computer-generated, trying huge numbers of alternatives until something breaks through, much like a rapidly mutating virus. The result, he said, is the offense has an overwhelming advantage in speed: “You have a highly automated attacker going up against systems that are still largely managed by people.”

But the right kinds of open architecture can allow a targeted system to change rapidly too, closing holes faster than the hackers can exploit them. “They’re designed to allow systems to be rapidly reconfigured,” Boyle said. “They allow you to change the identity of your system [at] speed.”

That is, they can if you do it right — but in open architecture, the devil’s definitely in the details.