EP21: Privacy in the Age of Agents with Niloofar Mireshghallah

Jan 7, 2026

Niloofar Mireshghallah, an incoming assistant professor at Carnegie Mellon University, dives into the intriguing world of AI privacy and model behavior. She discusses the surprising reliance of models on context over memorization and highlights modern privacy threats like aggregation and inference attacks. The conversation touches on linguistic colonialism in AI, the challenges faced by non-English languages, and the importance of academic research in preserving the nuances of learning and cultural representation. Niloofar calls for innovative AI tools for science and education while emphasizing the need for privacy-aware designs.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Mix Memory, Context, And Occasional Weight Updates

The ideal system mixes parametric memory, context, and online updates with periodic weight consolidation.
Purely external memories that never update internal weights will miss necessary learning and drift over time.

ADVICE

Use Stepwise Prompts And Human-in-Loop Checks

Use human-in-the-loop workflows that expose failures and guide the model through simpler precursor tasks.
Prompt models to solve simpler subproblems first so they perform robust analysis on the real task.

INSIGHT

Pre-Training Enables Creativity; Post-Training Polishes

Pre-training enriches core representations and helps creativity; post-training sharpens behavior for end users.
Relying only on post-training risks worse performance for non-English languages and niche scientific tasks.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Guest: Niloofar Mireshghallah (Incoming Assistant Professor at CMU, Member of Technical Staff at Humans and AI)

In this episode, we dive into AI privacy, frontier model capabilities, and why academia still matters.

We kick off by discussing GPT-5.2 and whether models rely more on parametric knowledge or context. Niloofar shares how reasoning models actually defer to context, even accepting obviously false information to "roll with it."

On privacy, Niloofar challenges conventional wisdom: memorization isn't the problem anymore. The real threats are aggregation attacks (finding someone's pet name in HTML metadata), inference attacks (models are expert geoguessers), and input-output leakage in agentic workflows.

We also explore linguistic colonialism in AI, or how models fail for non-English languages, sometimes inventing cultural traditions.

The episode wraps with a call for researchers to tackle problems industry ignores: AI for science, education tools that preserve the struggle of learning, and privacy-preserving collaboration between small local models and large commercial ones.

Timeline

[0:00] Intro

[1:03] GPT-5.2 first impressions and skepticism about the data cutoff claims

[4:17] Parametric vs. context memory—when do models trust training vs. the prompt?

[9:28] The messy problem of memory, weights, and online learning

[16:12] Tool use changes model behavior in unexpected ways

[17:15] OpenAI's "Advances in Sciences" paper and human-AI collaboration

[24:17] Why deep research is getting less useful

[28:17] Pre-training vs. post-training—which matters more?

[30:35] Non-English languages and AI failures

[33:23] Hilarious Farsi bugs: "I'll get back to you in a few days" and invented traditions

[37:56] Linguistic colonialism—ChatGPT changed how we write

[41:20] Why memorization isn't the real privacy threat

[47:14] The three actual privacy problems: inference, aggregation, input-output leakage

[54:33] Deep research stalking experiment—finding a cat's name in HTML

[1:01:13] Privacy solutions for agentic systems

[1:03:23] What Niloofar's excited about: AI for scientists, small models, niche problems

[1:08:31] AI for education without killing the learning process

[1:09:15] Closing: underrated life advice on health and sustainable habits

Music:

"Kid Kodi" — Blue Dot Sessions — via Free Music Archive — CC BY-NC 4.0.

"Palms Down" — Blue Dot Sessions — via Free Music Archive — CC BY-NC 4.0.

Changes: trimmed

About

The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.