Live Contract Trading Data Harvesting

Aug 1, 2019

The goal of this project was to build an ETL system collecting and computing data about trades and trading positions for various cryptocurrencies. The data about trades and trading positions were integrated on the fly from multiple trading platforms via REST APIs and web socket.

The data about trades and trading positions were fetched via multiple REST APIs and web sockets (depending on the trading platform). The data was then normalized, and some elements were aggregated, computed, and estimated. Finally, the normalized data were loaded into a common data model in a relational database (Exasol DB). 

The system architecture combined Apache NiFi native data processors and several custom data processors in Python.

The final data was used for financial analysis by an advisory company focusing on investments in crypto assets, in order to create investment recommendations for their clients.

Apache Nifi Python Exasol DB Linux Docker Different APIs Web Sockets
Artificial Intelligence | Data Science | Machine Learning | Big Data

Over two decades of comprehensive experience in data processing from BigData to AI