The intelligence community is developing a single cloud computing network to allow all its analysts to access and rapidly sift through massive volumes of data. When fully complete, this effort will create a pan-agency cloud, with organizations sharing many of the same computing resources and information. More importantly, the hope is the system will break down existing boundaries between agencies and change their insular cultures.

As in the rest of the federal government, lower costs and higher efficiency are the primary reasons for the intelligence world’s shift to cloud computing, said Charles Allen, formerly Under Secretary of Homeland Security for intelligence and analysis, currently a principal with the Chertoff Group, in an interview with Breaking Defense. Now in its eighth month, the goal of the effort is to connect the CIA’s existing cloud to a new cloud run by the National Security Agency. This NSA-run network consists of five other intelligence agencies and the FBI. Both of these clouds can interoperate, but the CIA has its own unique needs because it must work with human intelligence, which necessitates keeping its cloud slightly separate, he said.

The NSA’s cloud will incorporate the smaller organization-wide clouds of its partner agencies. One agency, the National Reconnaissance Office, which operates and manages the country’s spy satellites, has its own initiative to move its data to a cloud architecture. Jill Singer, the NRO’s chief information officer, recently noted that her agency’s cloud efforts plug directly into the larger intelligence community program. This allows vital satellite imagery to be shared with analysts across multiple intelligence agencies, she said.

One of the challenges of moving to a cloud-based architecture is that it creates a new business model for an organization, said Allen. This is important because in cloud computing, data does not stay in a single place, but migrates around different servers as an organization’s computing needs change. This saves money because it allows agencies to get more work out of fewer computers, but it also means they need to work out rules to share those computing resources with other organizations. Agencies need to determine what resources they need to share and then agree to rules and standards to work together. But he noted that getting to this end state will take a lot of engineering and rule-making.

There is a major inter-agency effort underway to lay the foundations for efficient resource sharing, which is an unprecedented level of cooperation for the intelligence community, Allen said. By comparison, he noted that it took the Defense Department’s various agencies 10 to 15 years to centralize its computer network resources. Parts of the intelligence community’s cloud effort have reached an initial operating capability, mostly the CIA and NSA portions, but he said the entire system won’t be fully operational for at least another five years.

Besides hammering out resource sharing policies, intelligence agencies will have to change their cultures to work more openly with each other. It will take strong and focused leadership from the Director of National Intelligence James Clapper and other top leadership to make sure agency cultures accept the new way of doing things, Allen said.

If the intelligence community’s cloud architecture is set up properly, it will make the nation much more responsive and flexible to new threats and changing situations. As it is, much of the military and civilian government already operates at a near-real time level in terms of collecting and reacting to information. “We’re operating both strategically and tactically at wire speed,” Allen said.

But there are additional issues that must be considered when moving to the cloud. Specifically, the data connectivity requirements will be huge, Allen said. The intelligence community already has a prodigious appetite for data of all kinds, from live streaming video from unmanned aerial vehicles to signal and data intercepts. All of this data must be collected, processed and analyzed quickly to make operational decisions, he said.

To process these massive volumes of data in near-real time, the intelligence community is turning to big-data, high-powered software analysis tools capable of searching for patterns and sniffing out specific types of information out of a sea of data. The community is also reaching out to industry to help develop new analysis tools and capabilities, because intelligence agencies can’t develop these tools on their own, he said.

The exact details of how and where agencies will share both resources and data are being worked out through the DNI’s CIO Office. Each agency will have its own configuration management board to help coordinate resource sharing, Allen said. The boards are helping to define the standards that will be needed to get the clouds up and running. “If you don’t, you don’t have a program,” he said.

Edited 9:45 am.