REALTIME data engineering for WINNING insights

From eCommerce to customer experience to workforce management, there are many time-series-related issues facing businesses today.  Making the BEST decision possible for your business means using the MOST current data, and in most cases, leveraging data changes right down to the millisecond.

Executive Summary

Modern data engineering is changing dramatically to keep up with the demands of an always-connected global environment, from designs based on past practices to designs supporting automated decision engines using prospective AI and introspective BI. Data engineering architectures must be loosely coupled for scaling, designed to handle data at the most granular level, accessing and processing only the data on demand needed for each business application that needs it. Data engineering architectures must remove the barriers between batch and streaming data. They must work for modern and legacy IT infrastructure without requiring complete overhalling.

CVX 3 is Cerebri AI’s 3rd generation platform. It was built over the past three years to address the most pressing data engineering issues in businesses looking to lead with a customer-focused data-centric view of the future.

CVX 3 checks out the ten critical factors when engineering data engineering for success.

1.Processing costs: – IT systems have morphed to make more and more automated decisions, provide more and more insight to support the many KPIs enterprises need.  The key to keeping processing costs as low as possible is to ensure that only the data required for insights and decisioning be used – no excess data baggage.  CVX 3 processes all data at the field level, which means only that data required for a decision is used. All the data presented to CVX 3 is persisted, a few mouse clicks away, to be used as needed, in our new low-code CVX Data Editor.

– Processing data at the field level for decision engines reduces processing overhead by as much as 75% versus at the record level.  Additionally, using a Pulsar messaging backbone versus Kafka  ( as presented in GigaOM June 2021) indicates that data processing savings could amount to an additional 80%.
– We believe CVX 3 is now the low-cost provider of data for automated decision engines ranging from the simplest BI tools to advanced AI models using deep neural networks.

2.Streaming: – Digital commerce drives business today. “ I’ll get back to you ” means “ I don’t want your business. ” Businesses must process both streaming data and batch data seamlessly for real-time decision-making. CVX 3 takes a novel approach of atomizing data at the field level and uses a publish/subscribe approach for data and insights that effectively turns batch data into streaming data to maximum performance.

3.Speed: – Decisions that use the latest up to the millisecond data updates are great. The decision engines they drive for both KPIs and Actions have to process, in a few seconds or less, all that data, all the time.  The CVX 3 platform has end-to-end software pipelines from our Data Source Transformers to our KPI insights and our Next Best Actions and Offers.  Removing the gaps between data engineering and modeling allows for real-time processing and real-time actions.

4.IoT: – The Internet of Things has invaded our lives. IoT data is invading the enterprise, be it telemetry pouring off vehicles and mobile devices or Internet routers from Cisco Systems and others.  IoT data sources are becoming critical for understanding customer behaviors, support issues, etc.  The volume of these data flows is often compared to drinking from multiple fire hoses; data is massive in quantity and scope.  CVX 3 provides for over 40 pre-written Data Transformers that enable aggregations and disaggregation of these critical data sources, making it simple to combine IoT data with traditional IT data, thus producing remarkable insights at IoT speeds.

5.Quality: – Quality monitoring must be real-time, from every facet of data input and transformation to decision making and execution.  CVX 3 embeds real-time monitoring for all transformers, models, and adjudication. It’s an alert system that keeps a close watch on the entire decision-making pipeline.

6.Results at scale: – Producing the BEST results can only be achieved by applying all the above critical factors at scale. Scaling has to be dynamic and adaptive. A loosely coupled data engineering and AI architecture allow a business to add, change and remove sources efficiently, to expand their decision engines across verticals without re-architecting at every juncture.  The CVX platform has been applied in dozens of use cases from customer experience & workforce management and other time-series related use cases.  The results have shown the amazing power of time-series, real-time, and data engineering done right. A 10 billion events pilot is now routine. Scaling is adapted to reduce the bottlenecks wherever and whenever they show up

7.Agile: One of the biggest frustrations in automating decision-making is finding then empowering the subject matter experts who understand the meaning of the data architecture used and why it resides here or there.  Legacy data engineering systems rely on joining (often massive) data tables over and over, then extracting views. This query-centric view is monolithic and difficult to approach without a software background. CVX 3 takes an entirely different approach. Rather than burying the business insights inside an inscrutable database structure, CVX 3 exposes insights in their raw form, then their refined form. The logic extracting insight is a flow of transformations managing all of the data and insight capturing/decision-making at the field level, connecting one Data Transformer to another as required. This insight-in-progress(“IIP”)  is to data engineering what work-in-progressis in manufacturing.  All insight-in-progress are persisted in CVX 3. Any aggregate or transformation can be subscribed to, transformed, and aggregated further for maximum agility in meeting business demands.

8.Security: – For CVX 3, data is encrypted as required by the business.  Personally identifiable information (“PII”) is masked by a data transformer, from ingestion right through to insights. Each CVX Data Transformer can be accessed and locked independently for security to prevent any unauthorized changes or access by downstream processes. Suppose changes are made to the logic in a CVX Data Transformer. In that case, there is a complete security protocol to ensure only those allowed to make changes do so and that the appropriate corporate staff authorizes such changes.

9.Auditability: – If you can’t audit, you can’t be secure.  CVX 3 has an end-to-end audit trail built-in. Data access, changes, errors, and similar details can be fully automated for real-time DevOps monitoring and use as required by the business.

10.Regulatory: – Cerebri AI works with all kinds of heavily regulated environments, such as central and retail banking, insurance, and telecommunications. Across these organizations and others, transparency and explainability of results are critical to eliminating bias and other regulatory concerns.  Replacing traditional manual and semi-automated BI decision engines with new AI decision engines requires auditable results traceable to the data elements that influenced the decision. CVX 3 provides for real-time explainability of AI predictions, using several advanced methods so our customers can monitor AI-driven results systematically.