Anomaly Detection in Business Intelligence Smart Metrics
The goal of this project was to develop an anomaly detection pipeline for business intelligence systems in order to trigger a BI report distribution - via slack or e-mail - when an anomalous situation was detected or send an alert to predefined groups of managers.
The system was designed to collect data online from various BI reporting systems (e.g., Tableau ). The data was later modelled as time series and stored in InfluxDB, following data was send to a stream processing engine called Kapacitor. Kapacitor was bridged with user defined methods responsible for indicating values that could be considered suspicious or anomalous, using both statistical methods and machine learning models. When the user defined functions detected anomalous value Kapacitor was triggering a REST API call to a back-end engine responsible to the final report distribution and alerting. Historical data stored in the time series database (InfluxDB) was also used to periodically compute the aforementioned statistics and train machine learning models.