LM-as-judge and multi-model evaluation

Many “AI startups” today are little more than thin wrappers around large language model APIs. But what happens when those APIs improve and the platforms absorb those features?

In this episode of The Ruby AI Podcast, Valentino Stoll and Joe talk with builder and investor Thomas Witt, founder of Vendis.ai and operator of the pre-seed firm Expedite Ventures. Thomas shares why he believes the next generation of durable companies must deliver real value deep in the product stack rather than bolting chat onto existing software.

The conversation explores why traditional CRMs are widely disliked and how an AI-native CRM might look completely different. Instead of rigid forms and required fields, Thomas describes a system where conversations themselves become the primary data source. Emails, meetings, and messages are embedded, searched semantically, and transformed into structured knowledge automatically.

They also dive into the architecture required to support this shift. From Ruby on Rails and Hotwire to DynamoDB, vector search, async Ruby, and multi-model LLM workflows, Thomas shares practical lessons from building AI-heavy production systems.

Along the way the discussion touches on agentic coding workflows, LLM-as-a-judge evaluation patterns, telemetry for prompt chains, and why small teams may soon replace the massive engineering orgs we’ve grown used to.

If you’re curious where Ruby, Rails, and AI systems are heading next, this conversation offers a fascinating glimpse.

Show Notes

Guest: Thomas Witt
Founder of Vendis.ai
Investor at Expedite Ventures

Topics we explore

• Why many AI startups are just “wrappers” around LLM APIs
 • What an AI-native CRM looks like when conversations become the database
• Why Thomas chose Ruby on Rails with minimal JavaScript using Hotwire and Stimulus
• Using Amazon DynamoDB instead of relational databases for AI workloads
• Hybrid keyword + vector search with OpenSearch and Elasticsearch
• Async Ruby patterns using fibers, the Async ecosystem, and the Falcon web server
• Orchestrating many concurrent LLM calls within a single user interaction
• Background job systems and queues such as Amazon SQS
• Code quality workflows with StandardRB and RuboCop
• Using models like Claude, OpenAI Codex, and Gemini together in multi-model workflows
• Observability and prompt tracing with Langfuse
• Why AI tooling may enable much smaller engineering teams

Mentioned in the Show

• Vendis.ai – Thomas’s AI-native CRM platform
• Hotwire – HTML-over-the-wire approach for modern Rails apps
• Falcon – Fiber-based Ruby web server
• Ruby AI Builders Discord – Community of Ruby developers building AI tools
• Chaos to the Rescue @ Artificial Ruby