ETL for Cryptoassets Transactions

The goal of this project was to build an ETL system collecting and aggregating data about various crypto assets.  The data about blocks and transactions on the crypto assets was fetched and integrated on-the-fly from multiple data sources via REST APIs.

The data was then normalized, and some elements were aggregated and computed. Finally, the normalized data were loaded into a common data model in a relational database (Exasol DB). 

The system was designed to run for a long-time with minimal supervision. It handles errors with checksums and counts verification. In case of a detected error in data if fetches and computes affected parts of data again. It handles network issues gently attempting to reconnect with a delay and exponential backoff strategy. The whole system was also designed with care to avoid overloading REST API servers (data providers) and to respect their fair use policies. 

The final data was used by financial analysts of an advisory company focusing on investments in crypto assets, in order to create investment recommendations for their clients.