Conceptual Data Modelling is the key to Success with Data Vault

The importance of using a conceptual model to map your business before venturing on your Data Vault project was the subject of Juha Korpela‘s presentation to the first Data Vault User Group meet-up of 2023 – titled ‘Capture Your Business Needs Within Conceptual Data Modelling.’. That applies whether starting from scratch or trying to upgrade your legacy systems. The Chief Product Officer at Ellie Technologies has spent more than 11 years in data management, business intelligence, data warehousing, data modelling and agile development practices across businesses in manufacturing, banking, and the public sector. Juha started by warning that any Data Vault implementation, regardless of how good the intention, risks failure if the data is not integrated in a way that provides value for the business. How to map your business to avoid failure? Juha explained that a Data Vault can integrate the data, and providing business keys identify the real business entities. They must also be valid across source systems. Using those business keys correctly will ensure you create an enterprise view of the data. Source systems have their own internal data models, but that may, or may not, match the actual business. There is no guarantee a table in a source system will correspond with a real business entity, or that a technical key field equates to a cross-business business key. At the start of your transformation project, it is essential to make a conceptual data model of your business. That requires your data architects to talk, and listen to the business users. Those business entities form a taxonomy. Juha summed up taxonomies as a hierarchical classification, explaining the “types of things” found in the enterprise. Adding relationships between those “things” results in an ontology, Ontologies explain the relationship between two or more “things”.

Why do some Data Vault projects fail?

Juha then demonstrated some examples to explain his previous points. In addition, he shared common reasons why Data Vault integrations sometimes cannot even make it beyond the first stage. In many cases according to Juha, they can range from:

• Not having an existing taxonomy or ontology

• IT departments reluctant to take on the task of creating them

• Over-reliance on data integration tools which create hubs, satellites, and links from input data

After looking at different ways source system-based solutions fail, Juha moved on to how they should look in a well-constructed Data Vault He stressed that the process should be technology-agnostic – in other words, not dependent on the databases or software tools used. That process can be summarised as the need to:

• Identify what things the business needs to know to succeed

• Use a conceptual data model (CDM) to capture the taxonomy and ontology after consulting the business experts

• Choose the right level of relationships between entities for the Conceptual Data Model

• Identify the REAL business keys before designing the core structure of the Data Vault logical model

• Automate away – but ONLY after mapping the designed model to the source systems

When drawing your data model, Juha stressed it must describe the real-life things in your business – events, places, people, and resources – with a real understanding of how those things interact and are related. Creating a business glossary between the data architects and the business experts will ensure a common language. Juha then quickly showed Ellie‘s technological solution that helps create a Conceptual Data Model, which leads to a logical Data Vault model.

Changes in source systems are managed by automation

In conclusion, Juha again warned that opting for a source-system-designed Data Vault is a simple option but will almost always end in failure. Only using Conceptual Data Modelling will capture the relationships that drive the business using a taxonomy and ontology of the real business entities – and does not require the data architects to look at the source systems. If there are changes to the source system, they can be managed through automation within the Data Vault. But if the business changes, modification to the Conceptual Data Model will be necessary – i.e., a manual process, but still likely to be a rare requirement. Once correctly modelled, automation will ensure the Data Vault can perform properly. That will enable the data analysts and data scientists to discover the real value within that data – and increase profitability through greater efficiency savings, and more sales. You can watch Juha Korpela‘ full presentation above.