Accuracy, rollout and RL feedback loops

The Derby Mill regulars host Shopify CEO Tobias Lütke on the heels of the ecommerce giant’s release of its Winter Edition 2026, aka The RenAIssance Edition, a significant artificial intelligence-enabled refresh of the company’s products and services. With more than 150 new and updated products, the update aims to help entrepreneurs, merchants and small businesses use AI to amplify their human creativity.

In one of the first conversations to happen with Tobi Lütke after the Shopify update, AI legends Rich Sutton, Sendhil Mullainathan, Niamh Gavin and Suzanne Gildert join Intrepid’s Ajay Agrawal to examine where artificial intelligence, machine learning and reinforcement learning may take ecommerce at the limit. How can AI help ecommerce merchants? What can machine learning do for small business? Could Shopify Sidekick’s agentic AI help merchants optimize their path to profitability? And will Shopify SimGym empower small businesses with the testing capability of much larger companies? It’s all on the agenda, and more, in our latest episode.

About Shopify CEO and co-founder Tobias Lütke:

Tobias Lütke is CEO and co-founder of Shopify, the marquee shopping cart system of the e-commerce industry, which he co-founded in 2006 after encountering difficulties trying to create an online snowboard retailer. Today, the company has a market capitalization of $210 billion USD, with customers in 175 countries around the world. FY2024 revenue was $8.88 billion US and transactions on the Shopify platform can amount to 10% of all US commerce.

GUESTS AND HOSTS

Tobias Lütke, CEO and co-founder, Shopify

Ajay Agrawal, co-founder and partner, Intrepid Growth Partners

Richard Sutton, senior advisor, Intrepid Growth Partners, 2024 Turing Award recipient, pioneer of reinforcement learning and professor, University of Alberta

Sendhil Mullainathan, senior advisor, Intrepid Growth Partners, MacArthur Genius grant recipient and professor, MIT

Niamh Gavin, senior advisor, Intrepid Growth Partners, Applied AI scientist and CEO, Emergent Platforms

Suzanne Gildert, founder and CEO, Nirvanic Consciousness Technologies

LINKS

Subscribe to The Derby Mill Series at our Substack (main site) or on YouTube, Spotify or Apple Podcasts.

Shopify’s Winter ‘26 Edition presentation and summary press release.

Mentioned in the pod: Susan Athey’s co-written journal paper is Artificial Intelligence, Competition, and Welfare, published by the National Bureau of Economic Research.

Derby Mill is created by the team at Intrepid Growth Partners and produced by Ghost Bureau.

DISCUSSION POINTS

00:00: Cold open with Shopify CEO Tobi Lütke saying, the goal is not to be the most powerful AI company, but to make AI gifts from labs maximally valuable to people.

01:01: Guest introductions, including Tobi Lütke, CEO of Shopify; Turing award winner Rich Sutton, who pioneered reinforcement learning; MacArthur Genius recipient Sendhil Mullainathan; applied AI scientist Niamh Gavin; and robotics and AI expert Suzanne Gildert.

01:50: Tobi discusses Shopify’s scale—operating close to six million storefronts and serving close to a billion customers purchasing about $30 billion a month in gross merchandise value.

03:17: Shopify as a counter-example to machine intelligence amplifying the power of large companies, instead using it to significantly boost smaller companies.

04:28: Toby Lütke provides an overview of Shopify, a Canadian company started 20 years ago that powers millions of merchants, often the websites customers buy from if it’s not Amazon.

06:57: Introduction of the three specific AI applications to be discussed, starting with “SimGym” for launching with confidence without real consumer testing.

07:38: Description of SimGym, a simulator with AI shoppers that predict customer behaviour and reflect the archetype of a merchant’s customers.

08:51: Discussion on the data backbone for SimGym’s personalized prediction, which includes transactional history, browsing behaviour matched to personas through standard clustering, and demographics from deliveries.

10:30: The goal of SimGym is to help small businesses get to conviction faster with their testing, as traditional AB testing takes a very long time for them.

12:58: Sendhil Mullainathan discusses how Shopify deploys scale economies to artisan producers, providing small businesses with data to make consequential decisions.

14:09: Introduction of “Sidekick,” Shopify’s agentic co-pilot, and the feature “Sidekick Pulse,” which delivers insights based on a store’s data, economic trends, and Shopify’s commerce knowledge, serving the non-sophisticated, time-and-money-constrained entrepreneur.

15:55: Sidekick is described as an assistive technology that automates tasks, finds factors to benchmark a business’s success, and provides insights in a human-like way, contrasting with the “very autistic” nature of typical software.

18:02: Shopify as a bridge between incredible research and the global network of commerce, bringing valuable morsels back to “entrepreneurship land”.

19:31: How merchants complained when Sidekick was temporarily taken down, with some referring to it as their “employee of the month”.

22:11: Shopify’s business model is fully aligned with customers; it does not charge for services like Sidekick because it benefits from bigger businesses, allowing the value of the AI to be absorbed in the existing model.

24:24: “Shop slop”—the concern that fully automated store production and drop shipping might push out small business owners.

25:48: Tobi Lütke argues that e-commerce is different from content generation because it has two governors: atoms must be assembled, and a transaction involves money, which is a rivalrous resource, meaning a purchase validates the value.

28:02: Rich Sutton asks how SimGym works and how it can be better than a shop owner’s intuition. Tobi Lütke’s response explains that it involves parameterized agents using a vision model browser loop to browse the website.

30:44: Discussion of Shopify’s advantage in having end-goal data (the sale) for its Reinforcement Learning (RL) system, providing true ground truth for the goal.

34:10: Ajay speculates that Shopify may become the most powerful AI company due to its access to vast data, the end goal (sale), and the large number of independent merchants, enabling a high degree of experimentation crucial for RL.

36:39: Introduction of “Shopify Product Network,” which uses machine intelligence to fill in product gaps for small merchants, like a skateboard store selling compatible helmets, thereby removing a scale economy disadvantage.

39:18: Introduction of the third AI product, “Sidekick Pulse,” which provides “next best action” predictions to merchant owners, advising on the most ROI- or sales-increasing action to take.

40:57: Niamh Gavin’s vision is that this technology enables a new age of affordable mass personalization by levelling the playing field for merchants and leveraging the community in a win-win network effect.

41:41: Suzanne Gildert questions the long-term objective function, asking if optimizing only for purchase volume could lead to a “dopamine addiction system” and suggests including consumer happiness.

42:34: Sendhil Mullainathan presents a positive future vision where Shopify’s architecture pushes AI in a different, decentralized direction, focusing on innovations that decision-makers (small merchants) find helpful.

44:32: Clarification of the two AI trajectories: autonomous decision-making (large organizations) versus human-machine “centaur” optimization (Shopify’s small merchants), where local information and the shop owner’s power are key.

47:40: The discussion notes that the centaur model would require a different set of performance benchmarks, focusing on improving human performance aided by AI.

49:54: Sendhil Mullainathan compares autonomous coding to co-pilots, suggesting that the centaur model focuses on making AI errors more transparent to humans and optimizing for diversity/variance rather than correctness.

53:38: Tobi Lütke reiterates that Sidekick and the other products function as “assistive technology with human in the loop,” aligning with the philosophical view that computers should work for humans and handle computing/data transfers.

56:08: Mention of the HSTU architecture, developed with Liquid AI and Nvidia, which has been “extremely game-changing.”

57:23: Rich Sutton discusses the limits of e-commerce, questioning whether decentralization should be around the merchant or the customer, suggesting a future where Shopify supports both.

01:02:30: The question is raised: what new and surprising thing will make online commerce different a year from now.

01:03:13: Niamh Gavin predicts that Sidekick Pulse’s ability to generate insights and automatically execute next best actions (like drafting a win-back email) will be the most surprising change in online commerce.

01:04:39: Suzanne Gildert expresses interest in consumers delegating agency to their own AI assistants, which could use simulation tools like SimGym to make buying choices from artisan merchants.

NUGGETS

Should AI Optimize for Correctness or Variance? (2101)

MacArthur Genius Sendhil Mullainathan to Shopify CEO Tobi Lütke: Should AI Optimize for Correctness or Variance?

Nugget 2 - The Most Powerful AI in the World (2102)

Could Shopify’s Winter ‘26 Edition Make It the World’s Most Powerful AI Company? Ajay Agrawal, Rich Sutton and Tobi Lütke discuss.

DISCLAIMER

The content of this podcast is for informational and educational purposes only and should not be construed as marketing, solicitation, or an offer to buy or sell any securities or investments. The opinions expressed in this video are those of the participants and do not necessarily reflect the views of Intrepid Growth Partners or its affiliates. Any discussion of specific companies, technologies, or industries is for illustrative purposes and does not constitute investment advice. Viewers are encouraged to consult with their own financial, legal, and tax advisors before making any investment decisions.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insights.intrepidgp.com