Custom Data Analytics Platform for API Provider
The client was facing trouble collecting and analyzing data on customers using their API platform. I was requested to research and develop a solution to enable data analytics and visualization on their platform. The delivered solution allowed my client to understand their clients’ behavior.
The client was using Kong API gateway in order to serve their API. Their attempts to collect data for analysis were targeted to use existing plugins sending data to various cloud-based platforms. They faced three kinds of problems:
- performance issues due to data collection overhead,
- lost data due to external provider connectivity issues,
- inability to visualize data to address business questions due to external platform limitations.
I designed a custom solution collecting data from Kong API Gateway and storing it in TimescaleDB (Postgres-based time-series database). Then, in Grafana, I created a set of dashboards precisely addressing business questions.
In order to collect data, I developed a custom data ingestion module processing data asynchronously to minimize performance footprint. The data was stored in an ultra-fast time-series database. Within the database, I created a set of continuous aggregates allowing to materialize the computed aggregates, while also providing means to continuously, and with low overhead, keep them up-to-date as the underlying source data changes. The continuous aggregates of different granularity and scope allowed to design of precise business dashboards visualizing data in real-time.
The final solution was deployed in production within the company Kubernetes cluster.
Part of this project was released publically:
