Trends in Data Engineering – Adrian Brudaru

27 snips

Mar 7, 2025

Adrian Brudaru, Co-founder of DLT Hub, shares his journey from corporate life to entrepreneurship in the realm of data engineering. He highlights the evolution of the modern data stack and the rising significance of open-source solutions. The conversation focuses on emerging trends like AI integration and innovative technologies such as Iceberg and DuckDB. Adrian also discusses the increasing opportunities for newcomers in the field and emphasizes the importance of understanding business contexts while navigating the evolving data landscape.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

dbt and Alternatives

dbt significantly improved data engineering by reducing boilerplate SQL code and promoting better practices.
Alternatives like SQLMesh focus on developer experience and efficient transformation code.

ADVICE

Workflow Orchestration Tool Selection

When choosing a workflow orchestrator, consider your team's familiarity and specific needs, not just popularity.
Airflow is a common choice, while Dagster and Prefect offer different advantages.

ADVICE

Focus on One Discipline

Focus on a single discipline at the start of your career instead of pursuing multiple data roles concurrently.
Gain clarity on your desired role and learn the necessary skills before branching out.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

In this podcast episode, we talked with Adrian Brudaru about the past, present and future of data engineering.

About the speaker:

Adrian Brudaru studied economics in Romania but soon got bored with how creative the industry was, and chose to go instead for the more factual side. He ended up in Berlin at the age of 25 and started a role as a business analyst. At the age of 30, he had enough of startups and decided to join a corporation, but quickly found out that it did not provide the challenge he wanted.

As going back to startups was not a desirable option either, he decided to postpone his decision by taking freelance work and has never looked back since. Five years later, he co-founded a company in the data space to try new things. This company is also looking to release open source tools to help democratize data engineering.

0:00 Introduction to DataTalks.Club

1:05 Discussing trends in data engineering with Adrian

2:03 Adrian's background and journey into data engineering

5:04 Growth and updates on Adrian's company, DLT Hub

9:05 Challenges and specialization in data engineering today

13:00 Opportunities for data engineers entering the field

15:00 The "Modern Data Stack" and its evolution

17:25 Emerging trends: AI integration and Iceberg technology

27:40 DuckDB and the emergence of portable, cost-effective data stacks

32:14 The rise and impact of dbt in data engineering

34:08 Alternatives to dbt: SQLMesh and others

35:25 Workflow orchestration tools: Airflow, Dagster, Prefect, and GitHub Actions

37:20 Audience questions: Career focus in data roles and AI engineering overlaps

39:00

The role of semantics in data and AI workflows

41:11 Focusing on learning concepts over tools when entering the field

45:15 Transitioning from backend to data engineering: challenges and opportunities

47:48 Current state of the data engineering job market in Europe and beyond

49:05 Introduction to Apache Iceberg, Delta, and Hudi file formats

50:40 Suitability of these formats for batch and streaming workloads

52:29 Tools for streaming: Kafka, SQS, and related trends

58:07 Building AI agents and enabling intelligent data applications

59:09Closing discussion on the place of tools like DBT in the ecosystem

🔗 CONNECT WITH ADRIAN BRUDARU

Linkedin - / data-team Website - https://adrian.brudaru.com/ 🔗 CONNECT WITH DataTalksClub

Join the community - https://datatalks.club/slack.html Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/... Check other upcoming events - https://lu.ma/dtc-events LinkedIn - /datatalks-club Twitter - /datatalksclub Website - https://datatalks.club/