Analyze Analyze Analyze — AI models
AI development has accelerated rapidly in the last 3 to 4 years. Back in 2017, very few knew what AI or deep learning specifically can do. What sort of data to collect ? what sort of model architecture to choose ? There was a fundamental doubt on the capability of this technology
Fast track to 2022, every business leader is cognizant of AI. Most of them have set up data science teams and most of them have the basic data necessary to kick start AI model development. Now, they are able to realize that AI model development is more complicated. Building a robust AI model that handles as many cases as possible — is a much more complicated problem.
We at Streamoid, faced very similar challenge few years ago. We were able to get the AI use cases very early and were also able to collect lot of data. So, we encountered the problem of building robust model very early on. The only way to solve this problem was to analyze the models. Understand where they were failing and correct them. More analysis leads to better models.
Analyze the edge cases — You can know more about the model on how it handles edge cases, unseen data and wrong data (data it is not meant to process) than how it handles normal cases. These edge cases are more powerful indicators of the model than normal data. So, we built a feedback loop that integrated to the end application and captured all those cases where the prediction was wrong. Where the model failed. This hard negative mining helped us capture new edge cases and build more robust models.
Analyze a model using traditional statistics — The traditional statistics are still the bedrock of modern AI techniques. Everything can be explained using statistics. Typically, when the problem does not fit a model statistically, it will not produce a robust model. So, by analyzing the data, models can be fine tuned. At Streamoid, we used TSNE plots, standard confusion matrices, PR scores and along with that violin plots and other out-of-distribution analyses.
Analyze the behavior of the models — AI models are like any other algorithms, we need to understand the boundaries in which they operate well. Some of the models might be based on deep learning hence its harder to understand. But, understanding the nature of the algorithm would be critical to its application. There are techniques like TracIn that explain model’s output based on the training data. Many visualization techniques that allow AI developers to understand bias. Many standard datasets that the AI model can be benchmarked against. For internal use cases, at Streamoid, we standardized the evaluation datasets and also integrated some of the visualization techniques in a no-code tool such that anyone can use it and understand the models.
AI Studio, the tool we built at Streamoid, helps in analyzing and explaining a model’s action in the most visual way. As a data scientist, you are supposed to constantly analyze your models. Treat your models like any other software wherein every wrong prediction is a bug. Only with this, AI will be successful in production.
Contact Streamoid Sales via malini@streamoid.ai or ashutosh@streamoid.ai for early access to the AI Studio tool.
