Real-time vs batch tradeoffs

Summary 
In this episode Himant Goyal, Senior Product Manager at Salesforce, talks about how data platform investments enable reliable, accurate metering for consumption-based business models. Himant explains why consumption turns operations into a real-time optimization problem spanning metering, cost attribution, billing, governance, and cross-functional ownership. He explores the richness required in usage data to support sophisticated pricing, the importance of treating metering like a financial system, and the architectural foundations - event schemas, durable ingestion, normalization/validation, a usage ledger, and clear serving layers - needed to power near-real-time visibility with fine-grained drilldowns. He also digs into anti-patterns and reliability concerns such as late or duplicate data, time zone pitfalls, SLAs, and automated policy decisions for pipeline failures. Himant shares practical guidance for capturing usage events from products and logs, balancing push vs. pull and real-time vs. batch processing to manage costs. He highlights configurable metering and rate-card versioning for rapid onboarding of new products, and the cultural shift required for finance, product, and engineering to co-own metering. 


Announcements 

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
• Your host is Tobias Macey and today I'm interviewing Himant Goyal about how data platform investments support consumption based business models

Announcements

• Introduction
• How did you get involved in managing the data products or data management?
• Can you start by outlining the types of businesses and products that are "consumption based" and the impact that it has on the economics of the company?
• What are the unique operational challenges that are presented by having consumption as the unit of cost?
  • How does the availability and accessibility of metering data impact the level of detail/nuance that the business can employ in their pricing strategies?
• When we talk about the infrastructure for usage tracking, it often feels like a high-stakes stream processing problem. What are the core architectural components required to build a reliable metering pipeline?
  • How do you think about the trade-offs between "push" models (application emits events) vs. "pull" models (the platform scrapes resource usage)?
• Accuracy is non-negotiable when data is tied directly to revenue. What are the strategies for ensuring idempotency and handling deduplication in the ingestion layer?
  • How do you address the "late-arriving data" problem in a usage-based world, especially when dealing with monthly billing cycles or credit exhaustion?
• From an uptime and reliability perspective, should the metering system be in the critical path of the service itself?
  • If the metering service is down, do you "fail open" and provide free service, or "fail closed" and impact availability? How do you build for that kind of resilience?
• One of the common pitfalls is treating metering like logging or observability. How do you ensure that usage metering is treated as a first-class product priority rather than an afterthought for the platform team?
  • What does the interface look like for product engineers to "register" a new billable event without breaking the downstream data contract?
• Once you have this data, there is often a requirement for real-time visibility for the end user. What are the data modeling requirements to support both "high-volume ingestion" and "low-latency querying" for customer-facing billing dashboards?
  • How do you bridge the gap between the raw event stream and the aggregated "billable unit" in the data warehouse or lakehouse?
• What are the most interesting, innovative, or unexpected ways that you have seen usage-based metering used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on building consumption-based data platforms?
• When is usage-based metering the wrong choice? (e.g., When does the complexity of the data platform outweigh the economic benefits?)
• What are your predictions for the future of consumption-based data architectures?

Contact Info
 

• LinkedIn

Parting Question
 

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links
 

• Hackernoon Post
• COGS == Cost of Good Sold
• Medallion Architecture

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA