Vanishing Gradients

LLM Architecture in 2026: What You Need to Know with Sebastian Raschka

83 snips
Apr 13, 2026
Guest
Sebastian Raschka
Sebastian Raschka, independent AI researcher and author of practical, code-first LLM guides. He digs into what modern model architectures actually contain. Conversations hit inference-scaling tricks, hybrid transformer/state-space designs, KV-cache and long-context tactics, Multi-head Latent Attention, and the tradeoffs of running local vs. frontier models.
Ask episode
AI Snips
Chapters
Books
Transcript
Episode notes
INSIGHT

Reasoning Is Often Distilled In Training Data

  • Reasoning behavior often exists in base models via examples in training data; chain-of-thought is more a behavior than a single technique.
  • Pre-training can already distill reasoning traces from post-trained models because training corpora include reasoning-style content.
ANECDOTE

Website CSS Fixes Feel Like Magic And Frustration

  • Sebastian uses LLMs to fix long-accumulated CSS cruft on his 12-year-old static website, and it often feels like magic when it succeeds.
  • When the model misaligns elements he resorts to screenshots and iterative fixes, then sometimes toggles to manual CSS tweaks.
ADVICE

Automate Writing QA With A Fixed Checklist

  • Automate tedious editorial and QA tasks with LLMs using a fixed checklist to save time.
  • Sebastian runs a ~20-item checklist (title casing, link checks, code-notebook sync) and asks the model to flag mismatches across notebook and docs.
Get the Snipd Podcast app to discover more snips from this episode
Get the app