AI Stories Build LLMs From Scratch with Sebastian Raschka #52
11 snips
Nov 21, 2024 
Sebastian Raschka, a Senior Staff Research Engineer at Lightning AI and bestselling author, dives into the art of building large language models. He shares insights on two significant open-source libraries, PyTorch Lightning and LitGPT, that enhance LLM training and deployment. The discussion shifts to his new book, where he outlines essential steps in LLM training and contrasts models like GPT-2 with the latest Llama 3. Sebastian also explores the universe of multimodal LLMs and their potential, highlighting exciting developments on the horizon.
AI Snips
Chapters
Books
Transcript
Episode notes
Leaving Academia For Hands On LLM Work
- Sebastian left academia because he enjoyed coding and wanted faster, more hands-on engineering challenges and easier access to large compute for experiments.
- At the university he was teaching ML repeatedly and found industry offered quicker iteration, large GPU access, and fewer managerial constraints.
Switch GPUs Without Losing Your Dev Environment
- Lightning AI provides a browser-accessible development studio that preserves your environment while switching between CPU and different GPU sizes to save cost and friction.
- This lets you debug on CPU locally then flip to many GPUs for training without reinstalling or re-uploading files, reducing wasted expensive GPU time.
PyTorch Lightning Cuts Multi GPU Boilerplate
- PyTorch Lightning is a thin wrapper that removes distributed-training boilerplate so code written for single GPU can scale to multi-GPU or multi-node with one-line changes.
- It also automates logging, checkpointing, mixed precision and strategies like FSDP or DeepSpeed for production-ready experiments.
