DataNation - Podcast for Data Engineers, Analysts and Scientists #59 – Apache Iceberg Catalogs (Nessie) vs Enterprise Data Catalogs (Colibra)
4 snips
Jun 25, 2024 Dive into the intriguing world of data catalogs as different frameworks are explored. Learn how enterprise data catalogs like Alation serve as knowledge bases, while Apache Iceberg catalogs optimize metadata for efficient querying. Discover the unique features of Iceberg, including versioning and branching capabilities that enhance data governance. The discussion highlights the convergence of governance features in both catalog types and the distinctions in their purposes for users and tools. Stay updated on the evolving catalog ecosystem and promising projects!
AI Snips
Chapters
Transcript
Episode notes
Enterprise Catalogs Serve People Not Engines
- Enterprise data catalogs (e.g., Alation, Collibra) store documentation, lineage, and access workflows for humans to discover and request datasets.
- They focus on visibility for people, not on providing query engines with direct access to data files.
Iceberg Catalogs Serve Query Engines
- Apache Iceberg catalogs let query engines find the correct table metadata (metadata.json) without expensive file listings.
- Their role is machine-focused discovery so engines can quickly read Iceberg tables.
Catalogs Point To The Right metadata.json
- Iceberg stores metadata layered over Parquet files and creates new metadata.json files on each change.
- Catalogs map tables to the current metadata.json so engines avoid costly S3 file listings and latency.