ToKCast Ep 255: Does this research explain how LLMs work?
47 snips
Jan 14, 2026 
Vishal Misra, a computer scientist known for his work on the 'Bayesian Attention Trilogy', joins to demystify language models. They discuss how LLMs work not through creativity but by mapping human explanations without true understanding. Misra argues these models, bound by their training data, lack the ability to innovate concepts or create new scientific knowledge. The conversation also touches on the limitations of Bayesian reasoning and the need for new architectures to achieve artificial general intelligence.
AI Snips
Chapters
Books
Transcript
Episode notes
LLMs Act As Closed Statistical Libraries
- LLMs are trained on vast curated libraries, making them closed systems for statistical reasoning.
- Brett Hall argues this closed nature enables mathematical probability methods to produce reliable next-token predictions.
Bayesian Maths Fits The LLM Environment
- In a closed training-library, probability calculus (including Bayes) becomes meaningful and computable.
- Brett Hall therefore accepts transformers can legitimately perform Bayesian-like calculations.
Bayesian Calculation Is Not Knowledge Creation
- If LLMs are Bayesian calculators, they do not create explanatory knowledge.
- Brett Hall concludes Bayesian updating recombines existing content but cannot invent truly new theories.
