Streaming Analytics & AI / ML centric business applications
Streaming Analytics Applications
Big data technologies have made distributed computing and massively parallel processing available to support processing of both structured and unstructured data – at scale. Now, there’s a shift towards getting actionable insights from the data using real-time analytics. For example, IoT systems have the immediate need for real-time analytics and operations. If the data is stale, fast response (although important) is not fully impactful. If the data is fresh, but people get to it slowly, many opportunities are lost, for example, critical equipment fails, fraud is committed or a customer buys from a competitor.
By 2020, 70% of organizations will adopt data streaming to enable real-time analytics.4
By 2020, 90% of IoT projects will adopt data streaming to enable analytics.4
By 2022, more than 10% of customer engagement hub architectures will include real-time event streaming and streaming analytics.6
Organizations want to use more real-time analytics to improve the accuracy and effectiveness of real-time decisions. Real-time analytics are becoming a necessary requirement for dealing with the new speed of business. Hence, organizations are investing in analytics to generate insights and turn insights into action: Descriptive (What happened?), Diagnostic (Why did it happen?), Predictive (What will happen?) and Prescriptive (What should I do?).
Real-time analytics requires two conditions: current data and predefined logic. For example an e-commerce application may determine the next best offer based on a real-time 360 degree view of customer formed on the basis of data in the customer data hub, available inventory, credit rating (external API), clickstream and customer location.
By 2020, more than 40% of all data analytics projects will relate to an aspect of customer experience.6
Technological innovations such as faster computation for streaming analytics including CEP, in-memory DBMS, cloud computing, massively parallel processing, Hadoop, Spark, NoSQL Database, GPUs and so on, are driving the development and running of decision services. The approach involves building models at design time and executing them at run time. The use of components such as a business rules engine for large or frequently modified rule sets and ML techniques to predict the future or estimate uncertain values are becoming more popular. Some of these ML algorithms can continuously retrain the model.
Prescriptive analytics are being used to find the best available solution with complex trade-offs. It involves optimizing an objective, subject to constraints, using common optimization techniques such as linear programming, non-linear programming, integer programming, and so on. It enables common scenarios that use such analytics include dynamic pricing and next best offer.
Stream analytics are being used for continuous intelligence and situation awareness where the input is a continuous stream of events from sensors, the web, corporate websites (clickstreams), copies of business transactions, etc. Common scenarios using such analytics include tracking customers’ locations and movements, detecting fraudulent activity, managing contact center service levels, etc.
There is a significant influence of open source and cloud computing technolgoies on real-time analytics applications. Companies are leveraging multiple open and closed products to implement such applications. Vendors are embedding open source in hybrid open / closed products. Such open source software includes Apache Spark and Spark Streaming, Apache Kafka, R, Python, and others. Additionally, cloud options are also proliferating with substantive offerings from Amazon, Google IBM Microsoft, Oracle, SAP, and Others.
ML / AI Driven Applications
Machine learning is making stream processing smarter. Conventional stream analytics comprised of rules, aggregations, pattern detection, predictive analytics and prescriptive analytics. The latest emerging trend in this area is dynamic or online machine learning that trains continuously as new data flows into the application.
The top-5 AI applications that organizations have integrated or plan to integrate with existing applications / solutions are: Customer Engagement Applications (34%), Call Center Service and Support (28%), Digital Marketing Platforms (23%), Cyberseurity (20%) and Financial Management Systems (16%). A significant majority (84%) are still in knowledge gathering / investigating / developing strategy or piloting AI solutions within the organization. Skills gap, identifying use cases, funding, and security / privacy concerns are the top challenges that organizations need to overcome for AI solutions adoption.
Conversational AI platforms (CAPs) are expected to be the next big paradigm shift in IT. Messaging-based applications are becoming the “new mobile home page” particularly for millenials. Consumers are paving a path that includes: interbot messaging, speech-to-text and text-to-speech, limited dialogue services and chatbots superseding apps. For example, for some consumers Facebook Messenger inbox has replaced their smartphone home screen. Additionally, there has been significant excitement around the potential of virtual customer assistants (VCAs) to delight customers while cutting call center costs.
By 2019, application functions based on artificial intelligence will be pervasive in 90% of enterprises globally.1
By 2020, 30% of all B2B companies will employ AI to augment at least one of their primary sales processes.6
Virtual Personal Assistants (VPAs) are becoming more integrated into the daily lives of customers. VPAs such as Apple’s Siri, Google’s Assistant, Amazon’s ALexa and Microsoft’s Cortana have gained popularity in recent years. These voice assistants, available on a variety of different devices, can help with a number of different tasks from driving directions to general questions.
There are many hosted AI Services being offered by leading technology vendors, these include:
· Natural Language Processing (NLP): Amazon Lex, Microsoft LUIS, Google Cloud Natural Language API, IBM Watson Natural Language Understanding, etc.
· Translation: IBM Watson Language Translator, Microsoft Translator, Google Cloud Translation API
· Speech to Text: Amazon Lex, IBM Watson Speech to Text, Microsoft Bing Speech API, etc.
· Text to Speech: Amazon Polly, Microsoft Bing Speech API, Microsoft Linguistic Analysis API, etc.
· Sentiment Analysis: IBM Watson Tone Analyzer, IBM Watson Personality Insights, Microsoft Text Analytics API
· Image Recognition: Amazon Rekognition, Microsoft Computer Vision API, Microsoft Face API, Google Cloud Vision API
· Knowledge Extraction: IVM Retrieve and Rank, IBM Watson Tradeoff Analytics, IBM Watson Discovery, Microssoft Knowledge Exploration, etc.
· Others: Google Cloud Jobs API, Google Cloud Video Intelligence API, Microsoft Bing Spell Check API, etc.
Additionally, leading vendors are also making ML more accessible to businesses by hosting services to support many common use cases such as:
· Virtual Assistants: Customer Service, Loan Applications, Product Availability, Returns Process, Product Information
· Predictive Analytics Actions: Account Planning, Risk Assessment, Investment Priorities, Recommended Content
· Image Recognition: Authentication, Parts Identification, Inventory of Assets, Real Estate Appraisal
By 2021, 10% of new development projects will have a developer-written test script and an AI-enabled agent optimizing test code.1
By 2021 30% of development teams will be using AI-generated code.1
“Conversational AI-first” will supersede “cloud-first, mobile-first” as the most important, high-level imperative for the next 10 years.2
As “tip of the iceberg” phenomena, chatbots, conversational and messaging-based applications can make most technologies (including legacy systems) more usable, transforming opaque tools that we have at our command into trusted, valued subordinates that participate in the conduct of our daily life and business.2
Conversational, AI-rich technologies will change how enterprises do business on the internet. In the long run chatbots will replace most mobile apps, as well as web applications. This transition is expected to start slowly and then accelerate, significantly.
In this blog, we explored some of the emerging analytics and ML – AI driven applications that need to be understood now for developing strategies for addressing the associated governance, risks and security requirements, comprehensively.
In subsequent blogs we will explore the impact of shifts in operating environments, and emerging integration solutions.
References:
1. Van L. Baker, Preparing your business applications for artificial intelligence, Gartner Application Strategies & Solutions Summit, 2017
2. Tom Austin, Mark Hung, & Magnus Revang, Conversational AI to shake up your technical and business worlds, Gartner, 2016
3. Whit Andrews, Where you should use artificial intelligence – and why, Gartner, 2017
4. Svetlana Sicular, Harness streaming data for Real-Time Analytics, Gartner, 2016