AI evals and human-in-the-loop evaluation

In this episode of DataTalks.Club, Paul Iusztin, founding AI engineer and author of the LLM Engineer’s Handbook, breaks down the transition from traditional software development to production-grade AI engineering.

We explore the essential skill stack for 2026, the shift from "PoC purgatory" to shipping real products, and why the future of the field belongs to the full-stack generalist.

You’ll learn about:

- Why the role is evolving into the "new software engineer" and how to own the full product lifecycle.

- Identifying when to use traditional ML (like XGBoost) over LLMs to avoid over-engineering.

- The architectural shift from fine-tuning to mastering data pipelines and semantic search.

- Reliable Agentic Workflows- How to use coding assistants like Claude and Cursor to act as an architect rather than just a coder.

- Why human-in-the-loop evaluation is the most critical bottleneck in shipping reliable AI.

- How to build a "Second Brain" portfolio project that proves your end-to-end engineering value.

Links:

- Course link: https: https://academy.towardsai.net/courses/agent-engineering?ref=b3ab31

- Decoding AI Magazine: https://www.decodingai.com/

TIMECODES:

00:00 From code to cars: Paul’s journey to AI

07:08 Deep learning and the autonomous driving challenge

12:09 The transition to global product engineering

15:13 Survival guide: Data science vs. AI engineering

22:29 The full-stack AI engineer skill stack

29:12 Mastering RAG and knowledge management

32:27 The generalist edge: Learning with AI

42:21 Technical pillars for shipping AI products

54:05 Portfolio secrets and the "second brain"

58:01 The future of the LLM engineer’s handbook

This talk is designed for software engineers, data scientists, and ML engineers looking to move beyond proof-of-concepts and master the engineering rigors of shipping AI products in a production environment.

It is particularly valuable for those aiming for founding or lead AI roles in startups.

Connect with Paul

- Linkedin - https://www.linkedin.com/in/pauliusztin/

- Website - https://www.pauliusztin.ai/

Connect with DataTalks.Club:

- Join the community - https://datatalks.club/slack.html

- Subscribe to our Google calendar to have all our events in your calendar

- https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ

- Check other upcoming events - https://lu.ma/dtc-events

- GitHub: https://github.com/DataTalksClub

- LinkedIn - https://www.linkedin.com/company/datatalks-club/

- Twitter - https://twitter.com/DataTalksClub

- Website - https://datatalks.club/