AI-assisted experimentation and A/B testing

🎙️ Ritish Chugh (Airbnb analytics engineering) joins Dietmar Fischer to unpack a problem almost every company has, but few name clearly: your metrics do not mean the same thing across teams. Finance, marketing, and sales can all talk about “revenue” and still end up in dashboard chaos. The result is wasted time, slow decisions, and leadership that does not fully trust analytics or AI.

In this episode, Ritish introduces the idea of the human data pipeline: the person who stitches together conflicting definitions, tribal knowledge, and unspoken assumptions just to answer basic business questions. Then we move into the fix: unified metric definitions, a data dictionary for business metrics, and a semantic layer that acts as a translator between raw data schemas and business meaning. That foundation is what makes natural language querying and conversational analytics viable at scale, without turning AI into a confident hallucination machine.

We also cover why AI adoption in analytics stalls when organizations prioritize models and infrastructure but neglect data quality, validation frameworks, and metrics governance. If you want AI to support decision-making, you need governed metrics, clear ownership, and a system that produces consistent answers across BI tools, SQL, and AI agents. Finally, Ritish shares wow moments from using AI tools to summarize years of code and PRs, generate deeper test coverage, and reduce time spent on manual SQL by building agents on top of a semantic layer.

📧💌📧

Tune in to get my thoughts and all episodes, don't forget to ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠subscribe to our Newsletter⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠: ⁠⁠⁠⁠beginnersguide.nl⁠⁠⁠⁠

📧💌📧

About Dietmar Fischer: Dietmar is a podcaster and AI marketer from Berlin. If you want to know how to get your AI or your digital marketing going, just contact him at argoberlin.com

Chapters

00:00 From data consulting to Airbnb and AI as a junior analyst

02:22 The human data pipeline and why metrics never match across departments

07:32 The fix: unified metric definitions, data dictionary, and the semantic layer translator

13:32 Why AI adoption stalls: data quality, trust, validation, and metrics governance

26:36 Data abundance, experimentation, and AI assisted A/B testing with humans in the loop

33:37 Wow moments with AI, role transformation, and why the Terminator is not invited (yet)

Quotes from the Episode

1. “AI just acts like a junior analyst, which is always available for you.”
2. “The first thing is… build that level of data definition that is unified for all.”
3. “No matter what AI models they’re using… if the data… is not up to the mark, it’s not going to give you the right results. It’s always going to hallucinate.”
4. “Every department has a different interpretation and definition of the metric.”
5. “I spend a lot of time really doing reconciliation between the numbers and data…”
6. “The most important thing happening is transformation…”

Where to find Ritish:

➡️ You connect with him on LinkedIn: linkedin.com/in/ritish-chugh/

📌 Keywords you’ll hear in action: semantic layer, data dictionary, metrics governance framework, unified metric definitions, governed metrics, natural language querying, conversational analytics, agentic analytics, data quality for AI adoption.

Music credit: "Modern Situations" by Unicorn Heads

Hosted on Acast. See acast.com/privacy for more information.