AI-Powered Search with Doug Turnbull and Trey Grainger [#133]

Doug Turnbull and Trey Grainger join the Weaviate Podcast to discuss all things AI-Powered Search! The conversation kicks off with designing search experiences, not all search queries are the same! Sometimes the user knows exactly what they want (a product ID, a specific file), other times they're exploring a broad category, and other times they need to compare and contrast options. AI is now making it possible to dynamically construct UIs around search results, moving toward what Trey describes as a "Minority Report"-style future where visualizations adapt on the fly to the query and the data.From there, the discussion dives into query understanding and domain modeling. Doug and Trey break down how LLMs can classify queries against existing taxonomies (like NAICS codes or Google's product taxonomy), while Trey explains a multi-tier RAG approach, using the index itself as grounding for query interpretation before executing the final retrieval. The conversation moves into agentic search, exploring whether iterative LLM-driven search loops reduce the need for ever-better embedding models, or whether simple tools like BM25 and grep are sufficient when paired with strong reasoning.Trey introduces wormhole vectors, a technique for traversing between sparse (lexical) and dense (semantic) vector spaces by treating query results as document sets with shared meaning, enabling exploration across vector spaces rather than treating them as orthogonal. The discussion also covers reflected intelligence, the idea of making search systems self-learning by mining user behavioral signals (clicks, purchases, skipped results) to continuously improve relevance through techniques like signals boosting, collaborative filtering, and learning to rank.The episode wraps with a conversation about how coding agents are changing the way Doug and Trey work, and Trey's philosophy of designing intentional agentic workflows with atomic agents rather than just handing an LLM a bag of tools.AI Powered Search (Discount Code = "weaviate")https://aipoweredsearch.com/live-course?promoCode=weaviate