Tool Use - AI Conversations How To Build a Hybrid AI System with Any-LLM (ft Nathan Brake)
17 snips
Mar 3, 2026 Nathan Brake, a Mozilla AI engineer and creator of Any-LLM, builds tools for talking to any LLM and hosting models locally. He discusses rapid model switching, running models on local hardware, cost and privacy benefits of small open-source models. He demos Any-LLM tooling, the Platform vault for keys and budgets, and his Porch Songs app that dogfoods model behavior.
AI Snips
Chapters
Transcript
Episode notes
Any-LLM Uses Provider SDKs To Reduce Fragility
- Any-LLM is a thin Python SDK that standardizes calls to many providers by delegating to each provider's official SDK for correctness and quick updates.
- Nathan built it to avoid brittle patches across integrations and to surface provider SDK breaking changes fast via integration tests.
Try Smaller Models Before Locking In Big Ones
- Regularly switch models during development to find smaller, cheaper models that meet your needs instead of defaulting to the biggest model.
- Nathan notes smaller models now can match last year's larger models, saving cost, latency, and enabling self-hosting.
Three-Part Stack With Privacy-Preserving Key Vault
- Any-LLM comprises three pieces: language SDKs, a gateway proxy for enterprise controls, and the Any-LLM Platform vault for encrypted key management and token tracking.
- The platform tracks usage and budgets without routing or seeing prompt content by encrypting user keys locally.