Big Gains Foreseen For Big Data In Military Applications

Big Data Investments

Big Data technologies include distributed computational systems, distributed file systems, massively parallel-processing (MPP) systems, and data mining based on grid computing, etc.

The US Department of Defense announced a FY 2017 budget request to invest in “Big Mechanism,” an effort to automate computational intelligence for biology, cyber, economics, social science, and military intelligence.

As per the budget, DARPA projects that its planned investment related to big data will rise from approximately $216M in FY 2015 to nearly $269M in FY 2017, a planned increase of 24.5%.

In 2014, Pentagon's high-tech research arm, the Defense Advanced Research Projects Agency (DARPA), announced a program it calls Mining and Understanding Software Enclaves (MUSE) to improve the quality of the military's software.

It wants to develop "big code" software packages capable of managing information without distorting it or collapsing under the weight of the large data sets, the so-called big data, which is at the heart of the Obama administration's technology programs announced in 2012.

Various Pentagon big data initiatives include research by the Defense Threat Reduction Agency to track dangerous pathogens, a DARPA-directed plan to mine health data to track cancer cells, a combination of data mining and patent analysis to track "disruptive technologies" that could alter the future of military equipment and planning and counter-force programs to track the spread of weapons of mass destruction by DTRA.

Big Data Tools & Technologies:

1.Apache Object Oriented Data Technology (OODT): Birthed at NASA's Jet Propulsion Laboratory, it uses structured XML-based capturing of the processing pipeline that can be understood and modified by non-programmers to create, edit, manage and provision workflow and task execution.

OODT also allows for remote execution of jobs on scalable computational infrastructures so that computational and data-intensive processing can be integrated into OODT’s data processing pipelines using cloud computing and high-performance computing environments.

Apache OODT is in use in many scientific data system projects in Earth science, planetary science, and astronomy at NASA, such as the Lunar Mapping and Modeling Project (LMMP), NPOESS Preparatory Project (NPP) Sounder PEATE Testbed, the Orbiting Carbon Observatory-2 (OCO-2) project, and the Soil Moisture Active Passive mission testbed. In addition, OODT is used for large-scale data management and data preparation tasks in the DARPA MEMEX and XDATA efforts.

2. Apache Bahir:  Bahir bolsters Big Data processing by serving as a home for existing connectors that initiated under Apache Spark, and provides additional extensions/plugins for other related distributed system, storage, and query execution systems.

3. Conditioned-based Maintenance (CBM): The US Air Force officials, data scientists, and other personnel are harnessing advanced data-handling and analytics tools, both software and hardware solutions, to streamline workflows, increase efficiencies and productivity, track and manage assets, and replace scheduled/time-based maintenance with conditioned-based maintenance (CBM) – saving at $1.5 million in one year.

Specifically, Air Force officials opted to use Teradata’s Aster platform to better manage Flightline maintenance; Inventory; and Aircraft depot data.

4. Hadoop software:  Hadoop was strictly a MapReduce batch analytics platform. Today, Hadoop can run a variety of workloads from batch to interactive and real-time.

5. Organizational, Relationship and Contact Analyzer (ORCA): Developed by U.S. Military Academy, this software was initially developed and used in military operations to identify networks of insurgents, and is now being used domestically. The military will use the domestic tests as a way to hone the software for future wartime applications, while in the meantime providing police gang units with a valuable free tool that could eventually see a more widespread deployment.

6. Tableau: It is a very effective tool to create interactive data visualizations very quickly. It is very simple and user-friendly, and can be used by anyone as this tool is designed to be used by developer as well as non-developers. You can also embed the visualizations on your website.

7. D3.js: D3. JS is a JavaScript library that is used for data visualization. To convert unstructured data collected from sources like government sources, social networking, eCommerce portals, etc. into some usable form or some sort of productive or understandable form, D3.JS is a very convenient tool. This JavaScript library is mainly used for creating data visualizations by highlighting web standards using CSS, SVG, and HTML, all with the full capabilities of latest browsers and no limitations of proprietary frameworks.

8. Power BI: Power BI allows developers to create visualizations and display data in a very accurate way that you will not find such facilities in other BI tools. It can be done with minimal efforts and using existing skills. The Power BI Desktop has a large variety of standard visualizations that include a range of reports that people usually needs. These illustrations can essentially be created through Power BI Desktop or the Power BI products. Though, there are always those conditions where a diverse style of graphs, charts or tables are required to get a specific desired result. These types of customizations can be developed internally or developer can retrieve other customized visuals from the Visualization Gallery.

9. R language: R is a language and an environment to run statistical calculations and produce data graphics. According to Google Trends, R is a language and tool very much related with creating data visualizations. It offers a variety of plotting devices where some of them are whole systems that a person needs to learn like a new language. 

Future Trends In Big Data for Defence:

As told by John Murphy, Head of Programs - Big Data Solutions Architect, US Department of Defence, Missile Defence Agency, who will be speaking at Global Big Data Defence summit 2017 on June 26.

1. Big Data Vs. Relational Data: Each use case or business case will require an understanding of where and how it is to be hosted. But the trend is away from traditional relational data structures and their supported infrastructure and towards general purpose big data architecture.
2. Data management concerns to prevent loss in value: With “Big Data” there is a need to apply more traditional data maturation and validation processes to ensure that optimum data value can be achieved.
3. Data Swamp V. Data Lake: Two approaches with big data have emerged.

The more rapid deployment and iteration cycles focus almost exclusively on using the data and information “as is” for analytics advocating for a more pristine and as close to the original data sources as possible. The resulting data set tends to become more of the “data swamp” environment with few rules on ingestion, quality and upkeep.

The second approach has elements of more traditional data management practices focusing on standardization, cleansing, auditability, consistency and over all managed care.

Big Data Analytics Firms: