“Organizations can no longer take a batch-based analytical approach to adaptation. They need to analyze, learn and predict in real-time, and they can do this with continuous intelligence. Continuous intelligence directly leads to increased business resiliency and agility, while enabling organizations to redefine competitive advantage,” says Simon Crosby, CTO, Swim, in an exclusive interview with EnterpriseTalk.
ET Bureau: How can a massive amount of historical data help enterprises in making decisions?
Simon Crosby: Continuous streams of events from “always-on” infrastructure and applications hold clues that organizations require to keep a competitive advantage, but most fail to find them in time. Sadly, it’s not getting easier. Over two million “smart” devices connect every hour, and there are a billion mobile devices in our hands. Organizations are drowning in streams of data and big-data “store-then-analyze” architectures that save and then process historical data struggle to find answers in time because data streams are boundless but data is only ephemerally useful.
So in answer to the question – is saving historical data even an option? Moreover, can any business afford not to adapt on-the-fly, based on the very best intelligence it can glean from the data streams that continuously arrive? The answer to both of these questions is “no.”
If we were to continue on our path, could existing application architectures built around databases keep up? Could we go back in time and reprocess all historical data later? Again, no. What we want is an efficient and analyzed form of historical data that is useful in the future. For example, the daily patterns of traffic at an intersection are more relevant than each car’s arrival time.
Continuous Intelligence delivers durable insights on-the-fly from streaming data, dynamically building models that drive analysis, learning, and prediction. Insights are necessary; what is critical though is that the “store then analyze” notion of applications has to change to an “analyze, react and then store” model, just to keep up.
ET Bureau: What is the biggest advantage of an open core platform for continuous intelligence at scale?
Simon Crosby: Open core platforms hope to make it easier for developers to create continuous intelligence applications. They use an open core model where they open source everything that developers need to quickly get from “too much data” to “hero.”
Streaming data builds the graph: as sources report their status, they are linked into the graph. Linked Web Agents see each other’s state changes in real-time. They continuously analyze, learn and predict from their own states and that of Web Agents they are linked to, and stream granular insights.
ET Bureau: What kind of business problems benefit from continuous intelligence?
Simon Crosby: Use cases include predicting failures on assembly lines and immediate traffic flow in cities. They also help derive moment-by-moment demand in power grids, detect hackers in critical infrastructure, and optimize connection quality for mobile handsets in 5G networks. A need-to-know-now strategy characterizes them, and they require real-time processing of streaming data. The concept of locality is key. Application insights must be timely and correct and delivered in the context in which they are most useful. For instance, a public transit application tracks and predicts arrival times of all buses in the city, and each user only cares about their own bus, its context and wants to know when their bus is approaching.
ET Bureau: How effective is the process of analyzing data before storing? Could you draw on some real cases?
Simon Crosby: A global telecommunications service provider needed to analyze data from millions of network elements and devices to identify service issues as they occurred. The provider’s data collection and analytics infrastructure took hours to find the source of network problems. An open core platform analyses 3-5 million network operational messages per second, totaling several petabytes of data per day.
From this live data, open core platforms can continuously generate insights about service impacts in milliseconds and predict the impact of a network outage in the immediate future.
The service provider has achieved significantly greater situational awareness and much improved service quality using Swim to measure and address network performance in minutes instead of hours.
ET Bureau: What future trends can enterprises expect from continuous intelligence?
Simon Crosby: Increasingly, for all businesses, customers are online and demanding informed and immediate responses to business. Business models are changing rapidly, and organizations that cannot respond in real-time are in peril. More than that, smart businesses learn and adapt continuously to anticipate demands or evolving customer needs. Organizations need to differentiate their offerings from competitors, adapt to the market shift, consumer behavior changes, and address customer needs. Furthermore, organizations can no longer take a batch-based analytical approach to adaptation. They need to analyze, learn and predict in real-time, and they can do this with continuous intelligence. Continuous intelligence directly leads to increased business resiliency and agility, while enabling organizations to redefine competitive advantage.
Simon Crosby is CTO at Swim, developer of the industry’s first open core platform for continuous intelligence at scale. He co-founded Bromium in 2010 and now serves as a strategic advisor. Previously, he was the CTO of the Data Center and Cloud Division at Citrix Systems and founder, CTO at XenSource. Simon holds a PhD from University of Cambridge.