Real-world evals and taste

Brian stresses large-scale, application-based evals and engineering taste to judge functional correctness versus intent.

Play episode from 25:29

chevron_right

Transcript

chevron_right

Transcript

Episode notes

Blitzy founders Brian and Sid break down how their “infinite code context” system lets AI autonomously complete over 80% of major enterprise software projects in days. They dive into their dynamic agent architecture, how they choose and cross-check different models, and why they prioritize advances in AI memory over fine-tuning. The conversation also covers their 20¢/line pricing model, the path to 99%+ autonomous project completion, and what this all means for the future software engineering job market.

Sponsors:

Blitzy:

Blitzy is the autonomous code generation platform that ingests millions of lines of code to accelerate enterprise software development by up to 5x with premium, spec-driven output. Schedule a strategy session with their AI solutions consultants at https://blitzy.com

Tasklet:

Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai

Serval:

Serval uses AI-powered automations to cut IT help desk tickets by more than 50%, freeing your team from repetitive tasks like password resets and onboarding. Book your free pilot and guarantee 50% help desk automation by week four at https://serval.com/cognitive

Claude

Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at ⁠⁠https://claude.ai/tcr⁠

CHAPTERS:

(00:00) About the Episode

(03:02) AGI effects without AGI

(07:07) Domain-specific context engineering

(16:54) Dynamic harness and evals (Part 1)

(17:00) Sponsors: Blitzy | Tasklet

(20:00) Dynamic harness and evals (Part 2)

(30:42) Graphs, RAG, and memory (Part 1)

(30:49) Sponsor: Serval

(32:26) Graphs, RAG, and memory (Part 2)

(41:17) Model zoo and memory

(50:07) Planning, scaling, and parallelism