CI, Automated Checks, and Data Quality

In this talk, Juan, Analytics Engineer and author of Fundamentals of Analytics Engineering share his professional journey from studying psychological research in Colombia to becoming one of the first analytics engineers in the Netherlands. We explore the evolution of the role, the shift toward engineering rigor in data modeling, and how the landscape of tools like dbt and Databricks is changing the way teams work.

You’ll learn about:

• The fundamental differences between traditional BI engineering and modern analytics engineering.
• How to bridge the gap between business stakeholders and technical data infrastructure.
• The technical "glue" that connects Python and SQL for robust data pipelines.
• The importance of automated testing (generic vs. singular tests) to prevent "silent" data failures.
• Strategies for modeling messy, fragmented source data into a unified "business reality."
• The current state of the "Lakehouse" paradigm and how it impacts storage and compute costs.
• Expert advice on navigating the dbt ecosystem and its emerging competitors.

Links:

• DE Course: https://github.com/DataTalksClub/data-engineering-zoomcamp
• Luma: https://luma.com/0uf7mmup

TIMECODES:

0:00 Juan’s psychological research and transition to data

4:36 Riding the wave: The early days of analytics engineering

7:56 Breaking down the gap between analysts and engineers

11:03 The art of turning business reality into clean data

16:25 Why data engineering is about safety, not just speed

20:53 Reimagining data modeling in the modern era

26:53 To split or not to split: Finding the right team roles

30:35 Python, SQL, and the technical toolkit for success

38:41 How to stop manually testing your data dashboards

46:34 Bringing software engineering rigor to data workflows

49:50 Must-read books and resources for mastering the craft

55:42 The future of dbt and the shifting tool landscape

1:00:29 Deciphering the lakehouse: Warehousing in the cloud

1:11:16 Pro-tips for starting your data engineering journey

1:14:40 The big debate: Databricks vs. Snowflake

1:18:28 Why every data professional needs a local community

This talk is designed for data analysts looking to level up their engineering skills, data engineers interested in the business-logic layer, and data leaders trying to structure their teams more effectively. It is particularly valuable for those preparing for the Data Engineering Zoomcamp or anyone looking to transition into an Analytics Engineering role.

Connect with Juan

• Linkedin - https://www.linkedin.com/in/jmperafan/
• Website - https://juanalytics.com/

Connect with DataTalks.Club:

• Join the community - https://datatalks.club/slack.html
• Subscribe to our Google calendar to have all our events in your calendar
• https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ- Check other upcoming events
• https://lu.ma/dtc-events- GitHub: https://github.com/DataTalksClub
• LinkedIn - https://www.linkedin.com/company/datatalks-club/
• Twitter - https://twitter.com/DataTalksClub
• Website - https://datatalks.club/