Live Contract Trading Data Harvesting

The goal of this project was to build an ETL system collecting and computing data about trades and trading positions for various cryptocurrencies from different trading platforms.

The data about trades and trading positions was fetched via multiple REST APIs and web sockets (depending on the trading platform). The data was then normalized, and some elements were aggregated, computed, and estimated. Finally, the normalized data was loaded into a common data model in a relational database (Exasol DB). 

The system architecture combined Apache NiFi native data processors and several custom data processors in Python.

The final data was used for financial analysis by an advisory company focusing on investments in cryptoassets, in order to create investment recommendations for their clients.

Marcin Wylot, PhD
Data Scientist & Machine Learning Engineer

20 years of experience in data processing from A to Z