Introducing From the Field: The Unconventional Guide to Cost Management

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Transcript

Corey: When you think about feature flags—and you should—you should also be thinking of LaunchDarkly. LaunchDarkly is a feature management platform that lets all your teams safely deliver and control software through feature flags. By separating code deployments from feature releases at massive scale—and small scale, too—LaunchDarkly enables you to innovate faster, increase developer happiness—which is more important than you’d think—and drive transformation throughout your organization. LaunchDarkly enables teams to modernize faster. Awesome companies have used them, large, small, and everything in between. Take a look at launchdarkly.com, and tell them that I sent you. My thanks again for their sponsorship of this episode.

Pete: Hello, and welcome to the AWS Morning Brief: Friday From the Field. Triple F; that's what we're calling it now. We’re going a new direction. I'm Pete Cheslock.

Jesse: I'm Jesse DeRose, and I'm so excited for Triple F.

Pete: Triple F. Hashtag Triple F. So, moving away, taking this into a new direction, we have… not stolen that's a little bit too aggressive. But we have been lovingly gifted this podcast from Corey Quinn after taking over while he was on paternity leave, we just kept on doing it; we never stopped, we never let him have it back. And he was nice enough just to give us this opportunity to take this Friday podcast into a new direction and talk about things that we're seeing as cloud economists in the field working with our clients.

Jesse: Yeah, it really started as this confessional discussion of weird architecture patterns that we've seen, but then it definitely morphed into more of the other things that we've seen from either our work with Duckbill or work with previous engagements or previous companies. So, it just felt fitting to rebrand just ever so slightly and focus more of our efforts on what are the things that we're seeing day-to-day? What are the major problems that our clients are seeing? What are some of the pain points we've seen? What are the new features from AWS that are really the interesting and important things to talk about?

Pete: Exactly. We have an interesting insight that I think a lot of folks in the industry don't get to see. We, for one, look at countless Amazon bills, seeing how people are spending their money. But we also are often reached out to directly to help engineering teams better answer questions that they're getting from finance. I mean, that's the biggest fear I have—

Jesse: Yeah.

Pete: —CFO comes walking over to my desk, and I haven't submitted an expense report recently like, what do they want?

Jesse: [laugh]. I didn't do it. It wasn't me.

Pete: Even worse is when some of your executives start learning some of these terms. And they say, “Hey, what's our cost per unit on Amazon Cloud?”

Jesse: Yeah, it is something that has morphed from just a conversation about engineering teams thinking about their architecture patterns and what might be best for them to getting the entire company involved—especially finance—to ask all these questions and really think about, what's the bottom line here? How can we better understand this cloud spend?

Pete: I know most people are probably thinking, “Doesn't tagging solve this problem. Can’t I just tag everything, and then I have all my answers, right?” Problem solved.

Jesse: I'm sorry, did you just tell me to go F myself there, Pete?

Pete: [laugh]. Obviously, we both know that even the best of companies, the most mature companies we work with, yeah, they might be about 90% plus fully tagged, but even those companies still have to put in a lot of effort to answer these questions and to understand where their spend is going. Because they say, that which gets measured gets improved. So, are you measuring your spend? Are you measuring your growth? Do you understand how your spend changes as usage changes, your customers change? I mean, there's countless questions. But there's another thing that we see, too, Jesse, right? This circle of pain, the—what is it—the cost management circle of pain.

Jesse: Yeah. Yeah. It's this really fascinating idea focusing on cloud cost optimization, where a company will realize that their cloud spend has gone up for whatever reasons, and they say, “Oh, no. We need to do something about this.” Whether that is because finance has come over and asked the question, or because engineering has caught the issue. 

And so they go through this quick session, maybe a quarter, maybe a couple months or more of figuring out, “How can we cut costs? Can we remove resources? Can we put these practices into place? Can we build some processes? Okay, now, everything's fine, right? We've managed to bring our costs back down. We managed to get rid of all of those EBS snapshots that were collecting dust and never to be used, so now we can go about business as usual again, right?” 

And so then they continue on as if nothing has happened. And without making long term changes, those costs are going to rise again. And then all of a sudden, we're back in the same spot of, “Oh, no, our cloud costs have gone up, why did they go up? We did all these things to make sure that we didn't have run into this issue again. Why are our cloud costs going up again?” And the cycle just repeats. It's a really unfortunate kind of spiral.

Pete: I remember my time at a startup where we were under a series of really high growth, a lot of customers coming on the platform. And my favorite meeting ever was the CEO talking about our financials. And he mentioned that our gross margin was negative 175%, which for the non-financial folks, means that for every dollar of income negative 175% is being spent for that. You normally want that number to be positive if you want to have a successful business. And remember, the line he said is, “We are going to successfully go out of business with a gross margin that is negative one hundred and seventy”—whatever I said. 

This is an important number that people need to think about. And what's amazing is that within a year, we had turned that around to be an extremely high gross margin because we started looking, and tracking, and bringing cultural change, and giving ownership to people to own these numbers. So, it's not just an engineering problem anymore. Everyone thinks that the Amazon bill is because your engineers built a certain thing, or turned on a certain type of instance. And sure, part of that is absolutely true, but I always like to say that your Amazo...