Contracts and Code: The Realities of AI Development

12 snips

Sep 23, 2025

Valentino and Joe dive into the reality behind AI salaries, discussing the disparity between hype and actual compensation. They debate whether any company can truly dominate the LLM market, emphasizing incremental improvements over clear winners. The hosts explore the complexities of benchmarking AI models and the necessity for customized evaluation tools. New OpenAI features that enhance prompt engineering are discussed, alongside the balance between playful experimentation and standardization in Ruby, highlighting its role in AI development.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Eval Tools Rarely Fit Out Of The Box

Eval tooling often falls short because products need custom metrics and calculators.
Teams end up rolling their own evaluation pipelines to measure the specific things they care about.

ADVICE

Prefer Native Tracing Or Build Bridges

Favor language-native tooling where possible for tracing and observability.
If SaaS tracing lacks Ruby support, be prepared to integrate or roll your own bridge for full observability.

INSIGHT

AI Products Break Traditional SaaS Economics

AI-driven products have high variable inference costs, so per-user economics look very different than classic SaaS.
Continuous or always-on workloads quickly amplify model costs and demand model-cost tradeoffs.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

In this episode, Valentino Stoll and Joe Leo unpack the widening gap between headline-grabbing AI salaries and the day-to-day realities of building sustainable AI products. From sports-style contracts stuffed with equity to the true cost of running large models, they explore why incremental gains often matter more than hype. The conversation dives into the messy art of benchmarking LLMs, the fresh evaluation tools emerging in the Ruby ecosystem, and new OpenAI features that change how prompts, tools, and reasoning tokens are handled. Along the way, they weigh the business math of switching models, debate standardisation versus playful experimentation in Ruby, and highlight frameworks like RubyLLM, Phoenix, and Leva that are reshaping how developers ship AI features.

Takeaways

The importance of marketing oneself in the tech industry.
Disparity in AI salaries reflects market demand and hype.
AI contracts often include equity, complicating true value assessment.
The AI race lacks clear winners, with incremental improvements across models.
User experience often outweighs model efficacy in AI products.
Prompt engineering is crucial for optimizing model performance.
Benchmarking AI models is complex and requires tailored evaluation sets.
Existing tools for AI evaluation are often insufficient for specific needs.
Cost analysis is critical when choosing AI models for business.
Incremental improvements in AI models may not meet user expectations. You can constrain tool outputs to specific grammars for flexibility.
Asking models to think out loud can enhance tool calls.
Reasoning tokens can be reused in subsequent AI calls.
Evaluating AI frameworks is crucial for business decisions.
Ruby's integration in AI is becoming more prominent.
The AI landscape is rapidly evolving, requiring adaptability.
Hype cycles can mislead developers about tool longevity.
Ruby offers a unique user experience for developers.
Tinkering with code fosters creativity and innovation.
The playful nature of Ruby can lead to unexpected insights.