Agile Building of Information Using Data Vault 2.0

Agile Building of Information Using Data Vault 2.0

What’s the Story? Data Glory…

Before you can use a Data Vault 2.0 implementation to gain the data analysis and insights business intelligence is able to bring to the table, you have to do certain things according to the widely regarded industry consultant, Bruce McCartney. Firstly, you have to ask the right questions of business leaders, about the aim and goals for a business intelligence project. And secondly, you have to create a story – one that encompasses what the business is about, its challenges, and where and what is its eventual destination. It started with a story about a balloon traveller who calls out to someone below and asks if they can tell them where they are, and where they are heading.

‘Are you a data professional?’

When the man answers by quoting the balloon’s exact position in longitude and latitude, the pilot assumes the passer-by must be a data professional, because his answer included more detail than he needed, and in a way that was of “no use to me at all.” To which the man on the ground replies, the balloonist must be a business executive because he admitted he had no clue about where he was, or where he was heading. He also relied on hot air to get to where he actually was, and only asked questions when he got into difficulty – before blaming the data team for his predicament. The clever metaphor captures conversations that happen in organisations around the world when it comes to why business intelligence projects can fail to meet their objectives.

Data Vault 2.0 delivers data analytics in an agile manner

Bruce, who has spent 30 years in the data warehouse industry, with 15 of those specialising as a fully-qualified Data Vault practitioner, is a firm believer that the benefits business intelligence is capable of delivering can be delivered in an agile manner, thanks to the advantages of

Data Vault 2.0. He stressed that a conceptual model is a must for any data warehouse, and the prerequisite to create a controlled vocabulary – in order to get the information out of the business you require. That requires an enterprise ontology, either a formal or informal one – you can learn more about that from Bruce’s presentation to the DVUG last year here. When carrying out consultancy work, Bruce establishes what the organisation’s or enterprise’s story is, and suggests creating a press release spelling out what future success would look like. And by using large language models like ChatGPT to support the role of the business analyst, you can quite quickly identify what the future could look like. So before you can establish what the goal has to be to take data and make better decisions from it, you have to question business leaders to uncover the value of business intelligence.

WHEN, HOW, WHERE, WHY AND WHAT

He also listed eight elements that make up a good, agile development story, largely arranged in a very simplistic order of When, How, Who, Where, Why and What. It is also important to establish the organisation’s existing levels of data literacy – is the business data centric or decision centric? Then you have to assess how mature the data model is. There are five typical levels. Most businesses are at level two (emerging) or three (intermediate) – where the value of data is recognised and investment in data management and governance has commenced, leading to a more structured approach with better data integration and data quality leading to more data-driven decision-making. Bruce then explained the traits and processes that denoted more mature data models. The analytic maturity model aims to establish four forms of business intelligence functions:

• Descriptive – what happened?

• Diagnostic – why did it happen?

• Predictive – what will happen?

• Prescriptive – what should we do?

Avoiding pitfalls of ‘Dark Data’

Once a company goes down the Data Vault path, it has the opportunity to operationalise its data analytics – especially with the ability to stream data in near real-time. This offers solutions in fraud detection, operational and policy risk, but such descriptive reports and dashboards require good data quality and correct mastering – avoiding duplicates and poor quality data that leads to incorrect assumptions and bad decisions. Bruce also warned there are many spurious correlations that can be drawn from data, and he also listed David J Hand’s 15 pitfalls from his book ‘Dark Data.’ covering the omissions and potential problems in collecting and storing data for analysis. Bruce then explained the nature of the optional business vault within a Data Vault 2.0 platform, which is the information delivery area – where the business comes for answers. Returning to the conceptual business model, which describes the “what” – not the who, how, where and when of the business – Bruce explained that the “things” are what the business needs to maintain records of in terms of its “elements” or “entities.”

Knowledge and wisdom via information delivery

Information delivery can provide knowledge and offer wisdom, by storing answers to the questions the business is asking – but not just by storing raw data, or all the transactions in a star schema, Bruce stressed. Those answers might come from KPIs – or from a set of features that might have to be inputted into a model to predict and proscribe what to do. Bruce ended by pointing out that another Data Vault User Group regular – John Giles – had produced taxonomy frameworks which could be used for mapping, as well as praising the online tool developed by ellie.ai. He also suggested a three-hour presentation from University of Amsterdam on YouTube on the subject.