Agile Data Warehousing / Business Intelligence: Addressing the hard problems

Scott Ambler – author and thought leader in agile data warehousing gives his thoughts

In today’s world, agile ways of thinking and working are at the heart of any large-scale business. This applies directly to the data world, too. In our latest meet-up, Scott Ambler – one of the world’s leading authorities in agile data – presented solutions to “hard problems” in agile data warehousing/ business intelligence. His presentation ensured members have plenty to think about as they look forward to a new year of obstacles and challenges. Scott started by defining ’agile data warehousing’ and ’business intelligence’ as “the act of providing quality information in a collaborate and evolutionary manner.” When trying to fathom how an organisation follows sound agile ways of thinking when it comes to data, Scott suggests seven simple steps: 1. Look beyond the data 2. Collaborate closely within your business 3. Be quality infected – poor data is the root of many ills 4. Embrace evolution – Rome wasn‘t built in a day 5. Be enterprise aware – find out what business users need 6. Make sure your project is fit for purpose 7. Ensure everyone works in an agile way While many organisations are conscious of doing things in short sprints,-Scott pulled no punches when it came to identifying “the Hard Problems,“ anyone trying to improve their data insights will have to deal with. Indeed he identified nine top problems and explored ways to mitigate each. The biggest upfront obstacle is that data architecture work takes too long, with pressure to deliver value in every sprint. The transformation requires the right infrastructure to be installed – and there are rarely enough agile data people to go round within one organisation, let alone the industry. According to Scott, the biggest elephant in the room is that most sprints are TOO SLOW! And it is a truism to say that meaningful data analytics often takes longer than a single sprint. When the product owner doesn‘t understand their own data, everything becomes a little more challenging. Finally, solving internal company politics is a must. It’s a very common challenge after years of Scott’s consultancy work - stakeholders within the business don‘t want to work with the data team. The good news, according to Scott, is that Data Vault 2.0’s methodology does a lot of the thinking to solve problems for you. By using a Data Vault 2.0, you can identify the main data sources as you try to understand the physical landscape. By modelling a conceptual diagram, you can not only identify the main business terms and their relationships, but you can also tie business terms to processes by mapping the data sources to the entity type. Returning to the issue of delivering value with every sprint, Scott recommends vertical slicing. This method means that that a working solution arises each time a problem is faced. That can be as simple as creating a new data element from just one, or even multiple sources, or changing existing reports and creating new ones. A new reporting view, or a new data mart table are all perfectly acceptable outcomes from one sprint. In terms of evolution, Scott said the key is to recognise that everything is not required straight away. The business‘s situation will change, as will it’s data analytics. You need to think about the future but biding your time will allow you to find the right solution at the right time. Big release value deliveries are almost invariably very late, over-budget, and risk being cancelled before they can even be delivered. While convenient for the IT team, the agile sprint will deliver value early on, and offer feedback to refine and improve desired outcomes. However, it requires the team to be skilled and disciplined. If data is the core of almost every business, then investing in people is the best way to have strong data analytics within it. Training, coaching and encouraging collaborative work is the only way to go for Scott. While data architects, data engineers, data scientists and data analysts all have critical roles to play, the more knowledgeable they can be across the data spectrum, the better. Scott also gave working examples of how to plan, design and execute an agile data project – evolving from the first release into a lean, continuous delivery process. That will require you to negotiate with stakeholders over their expectations. Active participation from stakeholders not only enables them to explore their own requirements, but it also makes the process enjoyable for them. Coaching the product owners by giving them the data knowledge they need will also pay off. Even so, stakeholders must make the time to help you help them for even the most agile project to deliver successfully. By shortening the sprint times, teams will be motivated to squeeze waste out of their ways of working, and move towards a continuous delivery strategy. Finally, the adage that rubbish in equals rubbish out must be tackled. Technical data debt is the biggest obstacle to leveraging information for informed decision-making, as well as increasing operational costs, and strangling an enterprise‘s ability to react to changes in the business market. Refactoring a database addresses those quality issues and not only improves design, but retains the behavioural and informational semantics. Scott‘s final thought was this… “The increasing pace of change, increasing complexity, and increasing volume of data demands nothing less than complete data agility.” You can watch Scott Ambler‘s full presentation above.