Platform
At Thanx in order to build data-driven products and power internal analytics, we’ve put together a modern data stack using the technologies described below. We follow an ELT approach for our data platform.
Stack#
Pipeline#
In our modern ELT approach to our data platform. Fivetran plays the role of the "Loader". It's sole responsibility is to copy data from a source data stores (database, API, etc) into a data warehouse. All this service contains is the configuration to access all the sources, the destination, and the frequency of how often data should be replicated.
Fivetran allows us to:
Automatically handle schema changes
Use it as a Snowplow collector (not currently is use)
Scale costs for loading data logarithmically — meaning that at high scale, pricing plateaus
FiveTran is managed exclusively through a web-based GUI, so there are no codebases to manage this. Access to this service is managed by the Platform, Merchant & Data Squad.
Warehouse#
A data warehouse is critical to a data platform. Data warehouses are designed for analytics (OLAP — online analytical processing), rather than transactions (OLTP — online transactional processing) like standard databases.
Snowflake allows us to:
Not depend on a single cloud infrastructure provider (AWS, GCP, Azure)
Automatically spin up and scale down virtual data warehouses on the fly
The snowflake configuration is fully managed by Terraform in the thanx-snowflake repository.
Modeling#
We use an opensource data modeling tool called dbt to run our transformations. The key value add of transformations is that the transformed data is clean and queryable. DBT allows us to manage modeling our data through a combination of SQL and Jinja, a common templating language used in the Python ecosystem. It also automatically builds a DAG for our models to manage dependencies between the models we define.
Our usage of dbt is captured in the thanx/thanx-dbt repository.
Business Intelligence#
Looker is designed for a variety of use-cases, including internal BI, external OEM’d, external embedded, and external API-proxy. Looker features a LookML modeling layer which allows use to have our changes be be versioned, reviewed, reused, and shared.
The goal of Looker is to be the single source-of-truth at Thanx where we go to find insights about our data. Our internal analytics will be powered by Looker and we will be powering the analytics in our merchant dashboard via Looker through API-proxied requests.
Our LookML configuration is managed in the thanx/thanx-looker repository.