In Part III of Cerebri AI’s series on Quality, we continue our exploration of the importance of Quality in artificial intelligence ( AI ) and machine learning ( ML ) to optimize business processes, increase revenue, and improve overall customer experience. At Cerebri AI, we group quality into four key disciplines:
These core disciplines must be mastered to deliver the best insights from AI/ML on a consistent basis. This whitepaper will focus on Models.
Cerebri AI helps large enterprises and organizations make better decisions by systematically using the computational power hosted in the cloud or on-premise high-performance clusters to apply methods and techniques at-times referred to as artificial intelligence (“AI”), or machine learning (“ML”).
In using machines to help make better decisions, we must systematically generate better insights ( data science ) to make predictions ( machine learning ) and ultimately taking the best actions ( artificial intelligence ) to achieve the desired results. We generally use the term AI to cover all three aspects of making decisions using machines.
Quality is always essential in any decision-making process. When using a systematic approach like that outlined above to make customer experience (“CX”) decisions, quality really, really matters. A bad insight or prediction in systematically taking customer actions may result in losing a customer, maybe even forever. Customers are almost always costly to acquire, easy to lose, and can be impossible to re-acquire.
Data-driven models have been used in business for decades to inform and guide strategic programs and operational excellence. AI modeling has supercharged many of the metrics we use in business today and have given us a chance at new growth opportunities. The increased use of AI models requires new quality metrics and benchmarks to audit the accuracy and integrity of algorithmic decisions made by those AI models. Model quality is as important as data quality as both are used to make decisions affecting customers. Similar to our previous whitepaper on data, models have to be monitored in real-time to ensure they continue to work according to pre-defined metrics.
Monitoring and validating AI model quality is required for both business performance and regulatory reasons. On the business side, we want the best possible insight or answer to make better decisions. On the other hand, regulatory compliance, such as the European Union’s General Data Protection Regulation (“GDPR”), or the CAN-SPAM Act of 2003, looks not at how well a model performs, but how it is used for communications.
In addition to these general regulations, many industries face additional scrutiny, such as banking, where algorithmic accountability is mandatory to ensure the validity of model results and prevent bias in decision making. Like Data, AI Models can experience drift in their predictive power when placed into production. Drift detection, coupled with a model lifecycle approach, is key to maintaining robust AI quality and performance over time. AI models are built and trained using a wide variety of machine learning techniques, resulting in an assortment of applicable models, but which model and what version should go into production? Defining the quality metrics for ML models that perform and meet regulations is not as straight forward as it is for rules-based decision engines. Quality is not cheap, but is well worth it when you get it right.
There are more than a dozen quality metrics for models, some with pretty arcane names including recall, precision, accuracy, F1, AOC, RMSE, K-S charts, to name a few. Each one measures a different quality or performance target which that is relevant and required depending on what CX metric is targeted. To understand more about the metrics involved in Model quality let’s look one step deeper at Model Validation in the following section.