Why Eval Frameworks Are Central

AI is pushing knowledge work toward a world where “leaders manage agents”—and eventually, where some management functions themselves are handled by AI. Shweta Kamble and Hari Iyer (founders of HaloVision) unpack that future and demo what it looks like today: an AI “third-party” that runs confidential 1:1-style conversations with employees, synthesizes themes into quantified “case files,” and creates a bidirectional channel between executives and the org.

00:00 - Intro

01:02 — “Undercover Boss” analogy: AI can surface ground-truth operational fixes at scale.

02:01 — “No ICs anymore”: the shift to managing armies of agents.

03:06 — AI can outperform average managers at listening, context, and coaching—at scale.

04:47 — Introducing Halo Vision: management + AI as a core intersection.

05:19 — What Halo does: confidential 1:1 conversations, analyzed into exec-ready insights.

06:00 — Key difference: not a suggestion box—Halo quantifies impact and outcomes.

06:36 — 1:1 controversy (e.g., “don’t do 1:1s”) and why time cost matters.

08:11 — Third-party confidentiality: why employees share more with Halo than internal tools.

09:30 — SurveyMonkey comparison: blending “survey + 1:1 + executive alignment.”

10:50 — Feedback loop requirement: employees must believe feedback leads to change.

12:06 — Founders’ backgrounds (Zoom AI/data products; CS/product design; Cisco ventures).

16:28 — Building Halo = “several companies in one”: auditing, privacy, PM estimation, infra.

18:03 — “Telephone game” across agents: why infra/evals matter for compound accuracy.

19:47 — Defining evals: correctness, reasoning tests, summarization/synthesis checks.

23:32 — Concrete eval example: summaries must trace back to transcript evidence.

27:03 — Added complexity: longitudinal context and time relevance (“6 months ago may not matter”).

30:39 — Prompt → context engineering: getting the right info to the model at the right time.

32:16 — Why off-the-shelf tools weren’t enough: auditability and tracing across abstraction layers.

37:18 — Live demo setup: Halo’s internal “case file” view with quantified issues.

38:01 — Example case files: exec jumping into low-level decisions; burn rate + delay cost estimates.

41:16 — Live call begins: confidentiality disclaimer + agenda choices.

41:50 — Halo’s questioning style: reflective, probing, tailored follow-ups.

46:17 — Positioning: Halo doesn’t replace 1:1s—it makes them more effective and focused.

47:00 — What they’re excited about next year: science/research advances + shifting human work.

Tools & technologies mentioned

Halo Vision — AI “third-party” that conducts confidential employee conversations, synthesizes insights into quantified exec recommendations, and helps align understanding across the org.

Evaluation frameworks (Evals) — Methods to test AI outputs (reasoning, summary accuracy, grounding) to prevent misleading conclusions and compounding errors in agent workflows.

LLM-as-a-judge — Using an LLM to grade another model’s output for correctness, grounding, or quality; often paired with other checks.T

racing / auditability / evidence links — Attaching each summary claim to specific transcript excerpts so you can prove where conclusions came from and debug errors.

Speech-to-text / transcription — Converting conversations into text artifacts that can be analyzed, summarized, and traced.

Fellow.ai — AI meeting assistant that joins meetings, summarizes, tracks actions/decisions, integrates with common work tools, and supports sensitive meetings with privacy/security controls.

Gemini (Google) — Mentioned as performing strongly for some use cases relative to other models at the time of recording.

GPT-4 / GPT-5 (and “5.2”) — Used as examples of model shifts affecting product behaviour (reasoning chains, tone/EQ, evaluation requirements).

Subscribe at⁠ thisnewway.com⁠ to get the step-by-step playbooks, tools, and workflows.