In this episode of The Cognitive Revolution, Nathan dives deep into the world of state space models with returning co-host Jason Meaux and special guest Quentin Anthony, Head of Model Training at Zyphra. Explore the cutting-edge Zamba 2-7b model, which combines selective state space and attention mechanisms. Uncover practical insights on model training, architectural choices, and the challenges of scaling AI. From learning schedules to hybrid architectures, loss metrics to context length extension, this technical discussion covers it all. Don't miss this in-depth conversation on the future of personalized, on-device AI.
Check out more about Zyphra and Jason Meaux here:
Zyphra's website: https://www.zyphra.com
Zamba2-7B Blog: https://www.zyphra.com/post/zamba2-7b
Zamba2 GitHub: https://github.com/Zyphra/Zamba2
Tree attention: https://www.zyphra.com/post/tree-attention-topology-aware-decoding-for-long-context-attention-on-gpu-clusters
Jason's Meaux Twitter: https://x.com/KamaraiCode
Jason's Meaux website: https://www.statespace.info
Be notified early when Turpentine's drops new publication: https://www.turpentine.co/exclusiveaccess
SPONSORS:
Weights & Biases RAG++: Advanced training for building production-ready RAG applications. Learn from experts to overcome LLM challenges, evaluate systematically, and integrate advanced features. Includes free Cohere credits. Visit https://wandb.me/cr to start the RAG++ course today.
Shopify: Shopify is the world's leading e-commerce platform, offering a market-leading checkout system and exclusive AI apps like Quikly. Nobody does selling better than Shopify. Get a $1 per month trial at https://shopify.com/cognitive
Notion: Notion offers powerful workflow and automation templates, perfect for streamlining processes and laying the groundwork for AI-driven automation. With Notion AI, you can search across thousands of documents from various platforms, generating highly relevant analysis and content tailored just for you - try it for free at https://notion.com/cognitiverevolution
LMNT: LMNT is a zero-sugar electrolyte drink mix that's redefining hydration and performance. Ideal for those who fast or anyone looking to optimize their electrolyte intake. Support the show and get a free sample pack with any purchase at https://drinklmnt.com/tcr
CHAPTERS:
(00:00:00) Teaser
(00:00:42) About the Show
(00:01:05) About the Episode
(00:03:09) Introducing Zyphra
(00:07:28) Personalization in AI
(00:12:48) State Space Models & Efficiency (Part 1)
(00:19:22) Sponsors: Weights & Biases RAG++ | Shopify
(00:21:26) State Space Models & Efficiency (Part 2)
(00:22:23) Dense Attention to Shared Attention
(00:29:41) Zyphra's Early Bet on Mamba (Part 1)
(00:33:18) Sponsors: Notion | LMNT
(00:36:00) Zyphra's Early Bet on Mamba (Part 2)
(00:37:22) Loss vs. Model Quality
(00:44:53) Emergence & Grokking
(00:50:06) Loss Landscapes & Convergence
(00:56:55) Sophia, Distillation & Secrets
(01:09:00) Competing with Big Tech
(01:23:50) The Future of Model Training
(01:30:02) Deep Dive into Zamba 1
(01:34:24) Zamba 2 and Mamba 2
(01:38:56) Context Extension & Memory
(01:44:04) Sequence Parallelism
(01:45:44) Zamba 2 Architecture
(01:53:57) Mamba Attention Hybrids
(02:00:00) Lock-in Effects
(02:05:32) Mamba Hybrids in Robotics
(02:07:07) Ease of Use & Compatibility
(02:12:10) Tree Attention vs. Ring Attention
(02:22:02) Zyphra's Vision & Goals
(02:23:57) Outro
SOCIAL LINKS:
Website: https://www.cognitiverevolution.ai
Twitter (Podcast): https://x.com/cogrev_podcast
Twitter (Nathan): https://x.com/labenz
LinkedIn: https://www.linkedin.com/in/nathanlabenz/
"The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis 