The Data T Data Contracts and Top Data Engineering Trends with Chad Sanderson
Jul 17, 2023
Chad Sanderson, Head of Data at Convoy and a leading voice in data engineering, dives into the realm of data contracts and their impact on data governance and quality. He explains how data contracts serve as essential agreements between data producers and consumers, addressing issues like schema drift and broken pipelines. Chad also highlights the importance of aligning data practices with business value, advocating for Git-style governance. Real-time data use cases and the relationship between contracts and data mesh are also explored, offering insightful perspectives for 2023.
AI Snips
Chapters
Transcript
Episode notes
Make Contracts Consumer-Driven
- Let consumers drive the contract: they specify the schema, semantics, and SLAs based on real use cases.
- Have producers sign off and provide tools to implement changes in their existing workflows.
Enable Easy Producer Ownership
- Give engineers context, simple tooling, and runbooks so they can own data quality without heavy friction.
- Integrate contracts into normal workflows (GitHub, SDKs) and provide clear remediation steps for failures.
Catalogs Complement Contracts
- Data catalogs complement contracts by enabling discovery of existing contracts, schemas, tables, and SLAs.
- Catalogs do not enforce contracts but help consumers find and validate data products.