Advanced Lakehouse Management With The LakeKeeper Iceberg REST Catalog

37 snips

Apr 21, 2025

Victor Kessler, co-founder of Vakama and developer of Lakekeeper, dives into the world of advanced lakehouse management with a focus on Apache Iceberg. He discusses the pivotal role of metadata in data actionability and the evolution of data catalogs. Victor highlights innovative features of Lakekeeper, like integration with OpenFGA for access control and its deployment using Rust on Kubernetes. He also addresses the challenges of migrating data catalogs and the importance of community involvement in open-source projects for better data management.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Catalogs in Multi-Cloud Ecosystems

Unity Catalog (Databricks) and Polaris (Snowflake) offer managed catalog services best suited inside their ecosystems.
Enterprises face multi-cloud, hybrid-cloud challenges needing agnostic catalogs focused on metadata, not just compute.

ADVICE

Centralize Authorization Controls

Centralize authorization management separate from compute and storage to avoid inconsistent data access.
Use identity providers (IDP) and integrate authorization with catalogs and compute via tools like OpenFGA and Open Policy Agent.

ADVICE

Lakekeeper's Modular Rust Design

Build Lakekeeper with Rust for performance and stability as a critical infrastructure piece.
Design it modularly with PostgreSQL storage and Kubernetes support for flexible deployment options.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Summary
In this episode of the Data Engineering Podcast Viktor Kessler, co-founder of Vakmo, talks about the architectural patterns in the lake house enabled by a fast and feature-rich Iceberg catalog. Viktor shares his journey from data warehouses to developing the open-source project, Lakekeeper, an Apache Iceberg REST catalog written in Rust that facilitates building lake houses with essential components like storage, compute, and catalog management. He discusses the importance of metadata in making data actionable, the evolution of data catalogs, and the challenges and innovations in the space, including integration with OpenFGA for fine-grained access control and managing data across formats and compute engines.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
Your host is Tobias Macey and today I'm interviewing Viktor Kessler about architectural patterns in the lakehouse that are unlocked by a fast and feature-rich Iceberg catalog

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what LakeKeeper is and the story behind it?
- What is the core of the problem that you are addressing?
There has been a lot of activity in the catalog space recently. What are the driving forces that have highlighted the need for a better metadata catalog in the data lake/distributed data ecosystem?
- How would you characterize the feature sets/problem spaces that different entrants are focused on addressing?
Iceberg as a table format has gained a lot of attention and adoption across the data ecosystem. The REST catalog format has opened the door for numerous implementations. What are the opportunities for innovation and improving user experience in that space?
What is the role of the catalog in managing security and governance? (AuthZ, auditing, etc.)
- What are the channels for propagating identity and permissions to compute engines? (how do you avoid head-scratching about permission denied situations)
Can you describe how LakeKeeper is implemented?
- How have the design and goals of the project changed since you first started working on it?
For someone who has an existing set of Iceberg tables and catalog, what does the migration process look like?
What new workflows or capabilities does LakeKeeper enable for data teams using Iceberg tables across one or more compute frameworks?
What are the most interesting, innovative, or unexpected ways that you have seen LakeKeeper used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on LakeKeeper?
When is LakeKeeper the wrong choice?
What do you have planned for the future of LakeKeeper?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA