Live Contract Trading Data Harvesting
The goal of this project was to build an ETL system collecting and computing data about trades and trading positions for various cryptocurrencies . The data about trades and trading positions were integrated on-the-fly from multiple trading platforms via REST APIs and web socket.
The data about trades and trading positions was fetched via multiple REST APIs and web sockets (depending on the trading platform). The data was then normalized, and some elements were aggregated, computed, and estimated. Finally, the normalized data was loaded into a common data model in a relational database (Exasol DB).
The system architecture combined Apache NiFi native data processors and several custom data processors in Python.
The final data was used for financial analysis by an advisory company focusing on investments in cryptoassets, in order to create investment recommendations for their clients.