The Role of Python in Shaping the Future of Data Platforms with DLT

23 snips

Oct 13, 2024

Adrian Broderieux and Marcin Rudolph, co-founders of DLT Hub, share their insights on the transformative role of Python in data platforms. They discuss DLT as a versatile library integrating with lakehouses and AI frameworks. The duo highlights high-performance libraries like PyArrow's impact on metadata management and parallel processing. They also explore the significance of interoperability and evolving governance challenges in data ingestion. Exciting plans for a portable data lake promise to enhance user access and experience in data management.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ADVICE

Improving Developer Experience with DLT

Use DuckDB for local development and onboarding, improving developer experience.
Standardize destinations and leverage Python generators for sources and reverse ETL.

INSIGHT

DLT's Data Interchange

DLT infers metadata, simplifying data movement; sources only emit data.
It leverages Parquet as the interchange protocol, enabling efficient data exchange.

INSIGHT

Portable Data Lakes

Portable data lakes are enabled by high-performance libraries, standardized formats, and portable query engines.
DuckDB and open table formats like Iceberg and Delta Lake are key components.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Summary
In this episode of the Data Engineering Podcast, Adrian Broderieux and Marcin Rudolph, co-founders of DLT Hub, delve into the principles guiding DLT's development, emphasizing its role as a library rather than a platform, and its integration with lakehouse architectures and AI application frameworks. The episode explores the impact of the Python ecosystem's growth on DLT, highlighting integrations with high-performance libraries and the benefits of Arrow and DuckDB. The episode concludes with a discussion on the future of DLT, including plans for a portable data lake and the importance of interoperability in data management tools.
Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Imagine catching data issues before they snowball into bigger problems. That’s what Datafold’s new Monitors do. With automatic monitoring for cross-database data diffs, schema changes, key metrics, and custom data tests, you can catch discrepancies and anomalies in real time, right at the source. Whether it’s maintaining data integrity or preventing costly mistakes, Datafold Monitors give you the visibility and control you need to keep your entire data stack running smoothly. Want to stop issues before they hit production? Learn more at dataengineeringpodcast.com/datafold today!
Your host is Tobias Macey and today I'm interviewing Adrian Brudaru and Marcin Rudolf, cofounders at dltHub, about the growth of dlt and the numerous ways that you can use it to address the complexities of data integration

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what dlt is and how it has evolved since we last spoke (September 2023)?
- What are the core principles that guide your work on dlt and dlthub?
You have taken a very opinionated stance against managed extract/load services. What are the shortcomings of those platforms, and when would you argue in their favor?
The landscape of data movement has undergone some interesting changes over the past year. Most notably, the growth of PyAirbyte and the rapid shifts around the needs of generative AI stacks (vector stores, unstructured data processing, etc.). How has that informed your product development and positioning?
- The Python ecosystem, and in particular data-oriented Python, has also undergone substantial evolution. What are the developments in the libraries and frameworks that you have been able to benefit from?
What are some of the notable investments that you have made in the developer experience for building dlt pipelines?
- How have the interfaces for source/destination development improved?
You recently published a post about the idea of a portable data lake. What are the missing pieces that would make that possible, and what are the developments/technologies that put that idea within reach?
What is your strategy for building a sustainable product on top of dlt?
- How does that strategy help to form a "virtuous cycle" of improving the open source foundation?
What are the most interesting, innovative, or unexpected ways that you have seen dlt used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on dlt?
When is dlt the wrong choice?
What do you have planned for the future of dlt/dlthub?

Contact Info

Adrian
- LinkedIn
Marcin
- LinkedIn

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA