top of page
Search

Federated Data Management

  • Writer: Rhys Hanscombe
    Rhys Hanscombe
  • Sep 10, 2024
  • 3 min read

At the 2024 Data Vault Conference, Chad Sanderson (CEO at Gable.ai, author of “Data Contracts” by O’Reilly, and founder of Data Quality Camp) delivered a compelling session on Federated Data Management. If you missed it, here’s your friendly recap—and a reminder to join our forum to keep the conversation going!


The Challenge: Data Developers Are Bottlenecked

  • 50:1 ratio: There are 50 software engineers for every data engineer.

  • 70% of data development time is spent on manual, non-scalable validation tasks.

  • 80% of incidents originate from unexpected upstream data quality issues.

  • Data developers are torn between high-ROI work (like AI, analytics, and experimentation) and urgent, manual work (like troubleshooting, impact analysis, and pipeline refactoring).


The Critical Questions

To build high-value data products (like AI), teams need to answer:

  • Where is the source of truth?

  • Can I trust it?

  • How do I access it?

  • Is it stable and durable?

  • Who is accountable?

  • What happens when data changes?

  • Who is using my data, and how?


But data producers and consumers rarely communicate directly, leading to confusion, duplicated data, and trust issues.


The Data Supply Chain: A Federated Model

Data management is like a supply chain—federated by design, with handoffs between producers and consumers. Each step requires visibility and quality. The output? A data product.

  • Data production is distributed across many teams and tools.

  • Data consumption is centralized, creating bottlenecks and ownership issues.

  • Result: Slow innovation, lack of trust, and recurring quality problems.


Federated Data Management: The Solution

Federated Data Management applies data management principles at both the source and destination, using a decentralized, automated approach. Its core components:

  1. Data Products & Data Contracts: Producers define what data they produce, how it should be used, and set SLAs/expectations.

  2. Discovery & Traceability at the Source: Catalogs are programmatically updated with end-to-end visibility on impact.

  3. Contract Enforcement & Governance: Data contracts are enforced at the moment of change, ensuring quality and accountability.


Data Contracts: The API for Data

A data contract is a data API—it defines the schema, semantics, SLAs, business logic, and policies that consumers expect. All data products must have a contract, and any changes follow a structured process. This enables:

  • Static code analysis

  • Metadata-based code review

  • Data integration testing

  • Live data monitoring

  • Automated enforcement and alerting


DevDataOps brings together DevOps, DataOps, and Data Engineering, meeting data producers where they already work—no need to change tools or workflows.


Real-World Impact

  • Companies are “shifting left” with contracts, catching issues early and building a culture of visibility and accountability.

  • Internal standards for data sharing pave the way for external standards and vendor integration.

  • Organizations like Netflix, PayPal, Uber, LinkedIn, and many more are adopting these practices to scale data quality and trust.


Key Takeaways

  • Federated Data Management is about decentralizing quality and governance, not just technology.

  • Data contracts are the foundation for trust, accountability, and scalable data products.

  • Automate and integrate: Meet teams where they are, and use automation to enforce standards and catch issues early.

  • Culture change: Visibility and collaboration are as important as technology.


Join the Data Community!

  • Sign up to our forum: Share your experiences, ask questions, and connect with fellow data enthusiasts.

  • Stay up to date: We host regular webinars, workshops, and meetups—don’t miss out!

  • Shape the future: Your feedback and participation help us build better tools and resources for everyone.


Final Thoughts

Federated Data Management is the future for organizations seeking scalable, high-quality, and trustworthy data. By embracing data contracts, automation, and a culture of collaboration, you can unlock the full value of your data supply chain. Ready to get started? Join the conversation, sign up for our next webinar, and let’s build the future of data together!

bottom of page