Why the world of data analytics and Data Vault is so excited by dbt?

Watch Drew Banin, co-founder of Fishtown Analytics the creators of dbt

The old fashioned way of using ETL to extract information from raw sources – already held in a data warehouse – and then transforming that data, on large costly servers, before being loaded back into the data warehouse, is living on borrowed time.

At least that would be the contention of Fishtown Analytics, creators of dbt. The development of the open source dbt tool is empowering data analysts and engineers’ ability to manage data pipelines and make most of business’s data analytics. Fishtown analytics co-founder Drew Banin – who developed dbt – gives an interesting insight into the latest developments in dbt, at the fourth of the UK Data Vault User Group’s virtual meetings. He explains how more and more businesses are empowering their workforce to get more from their data analytics using dbt to develop their data pipelines. Conforming source data to allow the many different reporting functions, which can change from day to day, requires a lot of logic to be applied. Drew goes through a series of sequences that show how the use of SQL can create data models that can be tested easily and depend on other data models, driven by metadata. dbt creates a directed acyclic graph (DAG) showing the data transformations required for your business intelligence operations and will run and test parts of the DAG. Watch this video to understand why there is so much excitement around dbt in the Data Engineering community.

At the end, Alex Higgs the lead developer of Datavault’s dbtvault tool, joined Drew for a Q&A. dbtvault is an open source package built on dbt which allows users to generate a Data Vault 2.0 data warehouse on a Snowflake database.   Author: ANDREW GRIFFIN