Himant Goyal, a Senior Product Manager who builds usage-driven data platforms, explains why metering should be treated like a financial system. He covers the architecture needed for reliable consumption tracking. Short takes dive into real-time versus batch tradeoffs, rate-card versioning, handling late or duplicate data, and the cross-functional shift required for finance, product, and engineering to co-own metering.
50:19
forum Ask episode
web_stories AI Snips
view_agenda Chapters
auto_awesome Transcript
info_circle Episode notes
insights INSIGHT
Metering Must Be Treated As A Financial System
Treat metering like a financial system where usage is the company's currency and accuracy matters as much as invoices.
Build event schemas, durable ingestion, normalization/validation, a usage ledger, and serving layers to support billing and analytics.
insights INSIGHT
Fine Grained Usage Enables Sophisticated Pricing
The richness of metering data sets the ceiling on pricing sophistication and what discounts or tiers you can offer.
Fine-grained events (per API call, per token, action context) enable simulations and proactive renewal conversations.
volunteer_activism ADVICE
Prioritize Real Time For Critical Usage Units
Identify which usage units need real-time visibility and which can tolerate batch processing to balance cost and responsiveness.
Deliver immediate top-line units (e.g., credits consumed) while processing detailed drilldowns (thousands of backend events) asynchronously.
Get the Snipd Podcast app to discover more snips from this episode
Summary In this episode Himant Goyal, Senior Product Manager at Salesforce, talks about how data platform investments enable reliable, accurate metering for consumption-based business models. Himant explains why consumption turns operations into a real-time optimization problem spanning metering, cost attribution, billing, governance, and cross-functional ownership. He explores the richness required in usage data to support sophisticated pricing, the importance of treating metering like a financial system, and the architectural foundations - event schemas, durable ingestion, normalization/validation, a usage ledger, and clear serving layers - needed to power near-real-time visibility with fine-grained drilldowns. He also digs into anti-patterns and reliability concerns such as late or duplicate data, time zone pitfalls, SLAs, and automated policy decisions for pipeline failures. Himant shares practical guidance for capturing usage events from products and logs, balancing push vs. pull and real-time vs. batch processing to manage costs. He highlights configurable metering and rate-card versioning for rapid onboarding of new products, and the cultural shift required for finance, product, and engineering to co-own metering.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
Your host is Tobias Macey and today I'm interviewing Himant Goyal about how data platform investments support consumption based business models
Announcements
Introduction
How did you get involved in managing the data products or data management?
Can you start by outlining the types of businesses and products that are "consumption based" and the impact that it has on the economics of the company?
What are the unique operational challenges that are presented by having consumption as the unit of cost?
How does the availability and accessibility of metering data impact the level of detail/nuance that the business can employ in their pricing strategies?
When we talk about the infrastructure for usage tracking, it often feels like a high-stakes stream processing problem. What are the core architectural components required to build a reliable metering pipeline?
How do you think about the trade-offs between "push" models (application emits events) vs. "pull" models (the platform scrapes resource usage)?
Accuracy is non-negotiable when data is tied directly to revenue. What are the strategies for ensuring idempotency and handling deduplication in the ingestion layer?
How do you address the "late-arriving data" problem in a usage-based world, especially when dealing with monthly billing cycles or credit exhaustion?
From an uptime and reliability perspective, should the metering system be in the critical path of the service itself?
If the metering service is down, do you "fail open" and provide free service, or "fail closed" and impact availability? How do you build for that kind of resilience?
One of the common pitfalls is treating metering like logging or observability. How do you ensure that usage metering is treated as a first-class product priority rather than an afterthought for the platform team?
What does the interface look like for product engineers to "register" a new billable event without breaking the downstream data contract?
Once you have this data, there is often a requirement for real-time visibility for the end user. What are the data modeling requirements to support both "high-volume ingestion" and "low-latency querying" for customer-facing billing dashboards?
How do you bridge the gap between the raw event stream and the aggregated "billable unit" in the data warehouse or lakehouse?
What are the most interesting, innovative, or unexpected ways that you have seen usage-based metering used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on building consumption-based data platforms?
When is usage-based metering the wrong choice? (e.g., When does the complexity of the data platform outweigh the economic benefits?)
What are your predictions for the future of consumption-based data architectures?
Data Engineering Podcast