In this article we configure an end-to-end data flow from data collection to dashboard visualization.
Specifically, we will showcase changes over time in currency conversion rates from the European Central Bank.
Every data project should start from the business needs.
For the purpose of this example, the business goal is to intelligently time the movement of cash between the eurozone and an external country (a common task for expats!). To do this, the end user would select the non-euro currency in a dashboard and get immediate insight into:
- What is the current exchange rate?
- What is the recent trend in the exchange rate?
- How does the current exchange rate compare to the historical exchange rate a week, month, and year ago?
Before we jump into implementation, we download a sample of the dataset and check some basics:
- Is the data sufficient to achieve the business goals? If it’s not, we need to understand this before spending resources.
- How clean is the data? Are there gaps or preprocessing that require a data pipeline?
This step takes anywhere from a few hours to a week depending on the complexity of the data. In this case it took only half a day due to the cleanliness and well-structured nature of the dataset. The analysis is available to view here.
It looks like the ECB data is well-structured, clean, and complete.
Given the project goal and our understanding of the available data, we can lay out the technical steps to achieve it. In a business context, this would be a good time to review existing data and data systems and incorporate them into the plan.
We look into the data source to understand the 3Vs – Volume, Velocity, and Variety – that help determine the required data architecture.
In our case, Velocity is once per day when the European Central Bank (ECB) publishes the latest rates. Volume and Variety are low - the ECB publishes a single well-structured 1.8 MB csv file that contains all historical data in the form of numbers and strings.
For this simple case we propose the following architecture:
- Database to hold conversion rates
- Web scraper that fetches and loads the latest rates from the ECB
- Dashboard to visualize the data
- Server to host the above components
To keep platform costs as low as possible, we stick to open source tools. We choose PostgreSQL for the database, Python for the web scraper, and Plotly for the dashboards. These tools are chosen because they are free, suitable for the task, and mutually compatible.
All the components will be deployed directly to a small EC2 instance in AWS which will act as the server.
In order to make installation, deployment, and maintenance simpler, we configure each of our services in its own Docker container.
Once the plan and results are agreed upon with stakeholders we can proceed with implementation.
Implementation can take anywhere from a week to months depending on the complexity of the project. For this project, I needed to set up an AWS account and secure EC2 instance from scratch which added time. However, the web scraper and dashboard were quite simple, leading to overall about a week of full time work.
Steps:
- Configure cloud computing account (AWS)
- Set up server (EC2 instance) with load balancer and firewall
- Set up Docker compose configuration with containers for the web scraper, database, and dashboard
- Program a web scraper and loader that works to load the latest data
- Implement the dashboard in Plotly
- Deploy the application to the server
- Schedule daily data pull and set up error alerts
- Monitor the application for at least a week
The end result is a consistently refreshing data flow that gives us quick insights into the strength of the Euro, and runs for the price of a couple fancy coffees every month. Link to the dashboard: https://currency-track.selvidata.com/
