In this special guest feature, Wes Caldwell, CTO of Intelligent Software Solutions, discusses how government-funded technologies successfully transition to the commercial sector, and how government grants play an important role in developing technologies that eventually make it to the commercial market – all serving to improve big data capabilities. Wes Caldwell is the Chief Technology Officer of Colorado Springs-based ISS, Inc. ISS develops sophisticated data visualization, event analysis, pattern detection, mission planning, and mobile software using net-centric and enterprise architectures. He oversees the company’s strategic technical direction and provides guidance in positioning ISS products and technology for maximum adoption.
We all know that defense budgets gave us the Internet and even today still yield fruits such as Apple’s Siri (a spinoff of a DARPA initiative). The synergy of innovation between the defense sector and private commerce will get a lot more intriguing as defense-generated IP starts being repackaged for Big Data applications in the private sector.
The log analytics space is a prime example: In large enterprises where web servers, database servers and other IT assets create gigabytes’ worth of logs to be monitored, prioritized and addressed, solutions with defense sector pedigrees are adding new value by being certified and plugged into enterprise search technology stacks.
Effective visualization to stop potential threats has been a consistent pain point. This is more important by the day in a world where the line between cybercrime and international espionage has blurred considerably. Being able to rapidly visualize activity from users (and their geographical locations), what processes they’re using and what assets they are accessing — and quickly drill down to spot granular, problematic activity — is vital to the enterprise’s IT watch dogs. Manually searching log files wastes critical time if you need to quickly spot and isolate atypical network activity.
Machine learning, natural language processing and enterprise search capabilities developed for defense applications are now helping corporate IT’s front line more quickly “shrink the haystack” of log events to find the needle of interest.
Natural language processing and entity extraction enriches the meaning of log files and can work in concert with machine learning to help a system administrator model and alert on abnormal behaviors outside the normal range of environmental chatter. Categorical exceptions can be more quickly prioritized and assessed by human operators. Logs can be “fuzzy matched” using Bayesian reasoning and provide probabilistic ways to determine whether or not logged events in the environment represent an impending threat.
All this can be bundled and affordably placed on top of open-source big data stacks to provide unprecedented visualization of terabytes worth of data, newly affordable power at scale, and formidable technology “co pilots” that make human analysts more effective at speed. In this instance, defense-centered vendors scored a “first”: Implementation of a certified visualization and data standard for a SOLR-based search platform.
Analysts and warfighters in theaters across the globe have an acute need to effectively organize, search, see and interpret massive amounts of data to isolate patterns and behaviors that, for example, may save soldiers’ lives by rapidly narrowing down the likely location of possible threats. It’s a high-stakes game whose byproduct enables corporate IT to do for the enterprise what defense sector solution providers do for troops on the ground in places like Afghanistan.
Each day, the private sector will be able to leverage more highly funded, production-proven IP wealth developed for defense. This cross-pollination will improve Big Data capabilities and outcomes everywhere it’s applied, creating more advantage and actionable insight for an exciting, rapidly evolving — and frequently dangerous — business technology world.
Sign up for the free insideBIGDATA newsletter.
Speak Your Mind