The Data Exchange with Ben Lorica Beyond the Chatbot: What Actually Works in Enterprise AI
13 snips
Sep 11, 2025 Jay Alammar, Director and Engineering Fellow at Cohere and co-author of "Hands-on Large Language Models," delves into enterprise AI. He discusses the challenges in understanding large language models and the adoption of GraphRag, emphasizing the gap between vendor enthusiasm and real-world application. Alammar highlights the balance between self-directed and collaborative learning in AI, and the critical role of evaluation processes for effective AI development. He also explores the potential of smaller AI models, showcasing their efficiency in addressing specific tasks.
AI Snips
Chapters
Books
Transcript
Episode notes
Onboard Data Sources With Metadata Snapshots
- Onboard new data sources with an explicit metadata snapshot that explains structure and conventions to the model.
- Provide tables, columns, tech stack, and test commands so the model can use context efficiently when interacting with that source.
Quality Retrieval Beats Huge Contexts
- Big context windows are costly and not always effective because models prioritize beginnings and ends of text.
- Precision retrieval plus well-chosen context beats stuffing million-token windows for most tasks.
Eval Data Is Strategic IP
- Treat evaluation datasets and processes as strategic intellectual property that define your product's capabilities.
- Build small internal test sets (10–100 examples) that reflect real inputs and outputs to guide model selection and regression tests.
