Last Week In AWS Podcast

AWS Morning Brief for the week of March 8, 2021 with Corey Quinn.

Links:Unconventional Guide to AWS Cost Management:https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/AWS Tagging Best Practices Whitepaper: https://docs.aws.amazon.com/whitepapers/latest/tagging-best-practices/welcome.htmlTranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I'm Pete Cheslock.Jesse: I'm Jesse DeRose.Pete: And we're back again, Jesse. We are back. But really have we gone anywhere to begin with?Jesse: We've been making our way slowly but surely through this Unconventional Guide. Lots of really interesting recommendations, lots of really interesting feedback from all of you, which we really, really appreciate. We can't wait to dive into some of those ideas deeper in future episodes.Pete: Yeah. And don't forget, you can give us additional feedback and questions at lastweekinaws.com/QA, feel free to add your name. Or not. Doesn't matter. It can be totally anonymous. That's fine with us. So today, we're talking about a topic that is very near and dear to our hearts. Jesse: Yes.Pete: It is tagging.Jesse: Yes.Pete: Tagging your resources in Amazon, or I mean really any cloud provider; any place you can tag something you probably should. And we're going to talk a little bit about strategies for that, how people use their tags, just all the fun things related to it. Tagging, it's easy to do, right, Jesse? You just tag your resources and all your problems go away.Jesse: Yep. Thanks, everybody, have a good night.Pete: So yeah, if you've enjoyed this podcast, please go to—no, I’m just kidding. Jesse: [laugh].Pete: Tagging is probably the thing that most companies are doing poorly, simply because it's hard, and it's an afterthought, and if you didn't have a really solid forced strategy to ensure tags and force compliance, you're probably not going back to fix it.Jesse: Yeah. It's not thought about as something that's a first-class citizen in the cloud world. When you think about the things that are important to your business model, you might think about getting your application out the door and running, maybe talking about business requirements for availability, failover, data retention, but tagging is nowhere on that list. That's not something that I think any organization thinks about as part of an MVP, let alone future iterations of their products.Pete: Tagging feels much like the same feeling I get when my doctor says that I should eat more veggies. Jesse: Oof.Pete: I know they're good for me; I know we need to do this. They have vitamins, and fiber, and all these wonderful things. But in order to make those veggies something I want to eat, we have to learn to make it more delicious. Personally, I find duck fat works to make them more delicious. I wish we could apply a duck fat strategy to the tagging problem.Jesse: Yeah, it's not an easy problem to solve. Or rather, I should say it is an easy problem to solve, but it's not something that anybody is quickly incentivized to solve. Tagging, just for the sake of tagging, it doesn't work.Pete: Yeah, it's that there really are no incentives for it. No good incentives. It's usually because someone came over to your desk and said, “Hey, what's this charge for? And who's using it? And what's the deal with this?” And you're going into Cost Explorer, and you're like, “Uh, I don't know. It's in this one account.” And that's as far as you can go to figure out who did what and why that thing is the way it is.Jesse: Yeah. There are so many different tagging strategies that we've seen. We've seen some clients talk about tagging as a way to potentially penalize engineers who aren't tagging or who are spending too much money. We've seen organizations who are tagging to reward teams that are tagging all their spend or keeping their spend optimized. Across the board, there are just so many different ways to go about this.Pete: So let's assume you are like most of the companies that we've seen. Definitely not all: there are some rare gems out there that are making tagging a long term and continual process, which we're actually going to talk about in a future episode, how to do that. But let's say you're just looking at your bill, you're looking at your usage, and you're saying to yourself, “Okay. I need to be better at this.” What do they say, “The journey of a thousand miles starts with a single step?” What is that first step?Jesse: Yeah there's a lot of different ways to go about this. I think there's a couple great places to start. Now, I will say AWS has a thrilling 24-page best practices white paper that we’ll throw a link in the [show notes 00:05:18]. Pete: Have you read that, Jesse?Jesse: I will say that I have read parts of it. I have not read all of it, and so I want to make it very, very clear to all of our listeners, this is not a document that needs to become the holy grail for your organization. I think in the same way that you could read the SRE book from Google and have some good takeaways, you can skim through this white paper, maybe read through a couple of the sections that seem most applicable to your organization, and then start with those ideas, start with those best practices, and then build them over time organically; develop them over time organically.Pete: I like to read it some nights when I'm just having trouble sleeping, and maybe by page two or three I’m just out.Jesse: Yeah. There's a lot of content in there talking about what to tag, why to tag. I think the best place for any organization to start is to think about what are the important things that we need to tag. And that's a conversation that's going to involve not just engineers, but also finance, potentially IT, maybe also security teams, depending on how your organization is built. Because ultimately, what you want to do is understand what are the things that my organization cares about when it comes to our cloud usage?&n...

Want to give your ears a break and read this as an article? You’re looking for this link.Never miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of March 1, 2021 with Corey Quinn.

Links:Unconventional Guide to AWS Cost Management: https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/Transcript Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: Ever notice how security tends to be one of those things that isn’t particularly welcoming to folks who don’t already have the word ‘security’ somewhere in their job title? Introducing our fix to that, Meanwhile in Security. To sign up for the newsletter or to find the podcast, visit meanwhileinsecurity.com. coming soon from The Duckbill Group.Pete: Hello, and welcome to Fridays From the Field. I'm Pete Cheslock.Jesse: I'm Jesse DeRose.Pete: And we're back, again. We're continuing our series, the Unconventional Guide to AWS Cost Management. And as always, if you have questions, as we are going through this series and want to learn more, go to lastweekinaws.com/QA. Thank you to all of those who have already submitted questions. Jesse: Yes.Pete: Really great ones coming in. Jesse: Thank you.Pete: We're going to take a couple of episodes in the future to answer those questions and really dive into them. So, keep them coming. We really love them so far. So Jesse, what are we talking about today?Jesse: Today, we're going to be talking about one of my favorite topics, which is that humans are the most expensive part of Cloud. Pete: Yeah, we hear this quite a bit. I mean, not just in salary, right? This is the line that usually is mentioned when we talk to folks about their Amazon spend. They say, “Well, outside of salary, Amazon is our most expensive bill.” Jesse: Yeah.Pete: That line has been repeated more times than I can count.Jesse: But what's so fascinating to me is that this really gets at the idea of total cost of ownership. I think that's ultimately what I really want to focus on for just a second. Total cost of ownership is thinking about all of the spend related to your cloud costs. Now, when you think about cloud costs, you will generally think about just the usage that you have within AWS, maybe some discounts from either an EDP or PPAs. But are you thinking about how much time it's taking your engineers to manage all of that usage, manage that infrastructure, manage the deployment pipelines that are living within the cloud? Are you thinking about all of those components and the cost of those components alongside your usage?Pete: Yeah, exactly. I think engineers are bad at this.Jesse: Yeah.Pete: Myself included. But this is something where we want to build things. That's why we're in this industry. And it's fun to build things. Maybe not so much fun to, kind of, ongoing manage those things. Looking at you, Cassandra and Elasticsearch clusters.Jesse: [laugh]. Yeah, it's this idea that there are definitely opportunities for engineers to spin things up and manage things on their own when you want to build that Kubernetes cluster and learn how to manage a Kubernetes cluster, learn how to build a Kubernetes cluster. That's great. We don't want to stop you from building and learning at all. But when you're building infrastructure for your organization, for your teams, for your products, is it going to be more cost-effective for you to build this solution yourself, or is it going to be more cost-effective for you to leverage existing managed services within the cloud?Pete: I like to call it operational FOMO, you know, the fear of missing out. And I think a lot of engineers suffer that when it comes to the new hotness, the new stuff. Kubernetes is a great example. I mean, I feel like a lot of those people were also equally like, “OpenStack is going to be the best thing ever.” And then it didn't. But I like to think of my time at a previous company where we deployed into the Cloud, specifically Amazon, and there was a fear that was, again, we've mentioned this before, it's an irrational fear about vendor lock-in. And that fear forced us into building forced us only using core primitives: S3, EC2, EBS, really. We really didn't use much more than that. I mean, obviously, the networks and stuff go in there. And the idea was, is that oh, well, we have this portability. And we—Duckbill Group, Corey, we've all talked about it, written about this. It's a fallacy. You're locked in for a lot of other reasons that I'm not going to go into right now. But because of that, we became very good at running our own databases and specifically consuming a large amount of time-series data. It was a security event application. And so one of the interesting flip sides of this outcome is that we ran our own monitoring infrastructure. I didn't pay for Datadog. They called me every single day and I was like, “My metrics infrastructure cost me $1,000 a month. You're going to charge me $50,000 a month. Even if you discounted that by half, I still am going to pay a lot more.” And the reality was, is that we became so good at managing these systems, we didn't need those services. But I always think back at like, at what cost? How much more time could we have invested in the application, the product, how we deployed it, availability, all that stuff, if we hadn't had to invest so much time into running our own Elasticsearch, running our own Mongo, our own Redis, our own Cassandra? We spent a lot of time doing those things.Jesse: Yeah, there's a lot of opportunities to leverage managed solutions for those things. Because, again, part of it is this idea of your engineers don't have to spend time managing this infrastructure; they can spend time on other things. But also think about what are the other cost components of this architecture that you may be able to leverage by using a native or a managed AWS service? For example, if you look at Amazon Elasticsearch—is it ‘Amazon Elasticsearch?’ Is it—Pete: I always forget if it's ‘Amazon Elasticsearch’ or ‘AWS Elasticsearch.’ And oftentimes, it doesn't feel like a rhyme or reason why they name it the way they do.Jesse: Well, let me put it this way. If you look at the managed Elasticsearch service on AWS, you don't end up paying for some of the things that you might pay for if you were managing that infrastructure yourself, like data t...

Want to give your ears a break and read this as an article? You’re looking for this link.Never miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of February 22, 2021. with Corey Quinn.

Links:Unconventional Guide to AWS Cost Management: https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Pete: Hello, and welcome to the AWS Morning Brief. I’m Pete Cheslock.Jesse: I'm Jesse DeRose.Pete: Fridays From the Field, Jesse. We're back again.Jesse: Back, back, back again.Pete: I always say that when I rage quit computers, it would be fun to be a farmer. And so maybe this is a little trial run “Fridays From the Field.” I'm just out in the field.Jesse: So basically, what I'm hearing is that you are the old man out in the field, yelling at the clouds as they go by.Pete: Well, now that I work from home pretty much all the time as part of Duckbill, but also due to COVID. I do yell at the squirrels who constantly tear up my yard. I've now turned into that person.Jesse: [laugh]. Oh, oh, Pete, I'm so sorry.Pete: Those squirrels. I hate them. So we're back again, talking about the Unconventional Guide to AWS Cost Savings. And this time, we're talking about ‘infrastructure code smell.’Jesse: Ooh, fun one.Pete: I like to equate this to, who brought the fish for lunch and microwave to that?Jesse: I always understood that at a deep core level, but didn't really think about it until I actually did microwave fish one day, and I regret everything.Pete: Don't do it. I'm telling you, folks, don't do it. You can bring tuna fish in. I guess that's fine. That's a little bit better. If it's packed in oil, it actually is a lot less smelly. Should we do a food podcast? No, I’m just kidding. [laugh].Jesse: [laugh].Pete: So ‘code smell,’ I do want to bring this one up because I actually did a little bit of a TIL—today I learned—with code smell. Yeah, this term was actually coined by someone that was a writer about the agile software movement, Kent Beck. He was working with Martin Fowler, who's a noted author about programming. In the book called Refactoring, they coined this phrase ‘code smell.’Jesse: I did not know this.Pete: Yeah. You know, you kind of hear a term, you just accept it without really understanding why. But what it was called in this book was, code smell is a surface indication that usually corresponds to a deeper problem in the system. So obviously, it is what it sounds like: something smells. Something doesn't seem good here. And obviously, it can take a lot of forms. You most often hear it in, obviously, software engineering but, guess what? Software engineering has expanded to manage our infrastructure, right?Jesse: Mm-hm, absolutely. Yeah, it's not just about—or I should say, infrastructure smell is not just about wasted resources. It's really thinking about all of those one-off hacks that got you this far. So, that one time that you couldn't deploy something into production, so you just said, “You know what? I'm just going to log into the console and spin up that instance, and then call it a day, and close the change order, and be done with it so I don't have to worry about it. Maybe I'll open a ticket to see if I can figure out what happened in the deployment pipeline, but I'm not going to worry about it.” All those little things that you did along the way that aren’t probably the best practices that you ultimately should be following and ultimately want everybody else to be following. Pete: Yeah, and I'm looking at you, software infrastructure manager, who is still running an m1.medium in production. That's code smell. Jesse: Oof.Pete: Anyway. Just don't use the m1.mediums. Let them go away. But, Jesse, you're right. It's not just those hacks and one-offs. It's kind of back to the context. It's the how. How you're doing certain things with these Amazon resources, right?Jesse: Yeah. And I think that's something that's a really important caveat, the call out because there is always a balance between premature optimization and waste. I struggle with this one a lot. My brain automatically thinks, “Well, if I'm going to do this, I'm going to do it the right way the first time, and I'm going to do it the streamlined automated way the first time so that I can just have it all set up the very first go, and set it and forget it and be done and walk away.” But in most cases, that's not how it works.Pete: Yeah, that is a complicated topic that I've struggled with as well. I've worked for predominantly unprofitable startups. We have a burn rate. We have only a certain amount of money in the bank and you divide by what your spend is, and that's when you're out of money. And doesn't necessarily mean the company's out of business, but it could mean that all that sweet equity that you have no chance of actually turning into real cash has even a less chance of turning into real cash. So, we often in the startup world make those decisions where we try to just get it done in what we hope is the best way possible. Again, we'll regret it two or three years later, but—Jesse: Regardless of the way you set it up the first time, we will regret it two or three years later.Pete: It's so true. Even if you say, “I’m going to set this up in the best way possible,” things change, and scale breaks everything eventually. So, in a couple of years, you're just going to be doing things in a different way—for better or worse—than you were doing. And it's kind of just all for not, in many cases.Jesse: One of my favorites that I see is application logs that are pushed into CloudWatch because you want to be able to see all of your logs in CloudWatch or all your metrics in CloudWatch. But then those same logs and metrics are then being sent off to Kinesis for analysis, they're being sent to Splunk for analysis, they're being sent to Datadog, or insert other third-party vendor here for analysis. So effectively, all you're doing is putting the data into CloudWatch as a cue to go to somewhere else. And CloudWatch isn't cheap. CloudWatch logs are expensive.Pete: Exactly. This is one of my most frustrating painful-to-see, dare I say anti-pattern of Amazon usage is, partly Amazon to blame on this one because they do make it so easy to get your logs into CloudWatch. It's a default option. If you turn on flow logs, you can have your flow logs go to CloudWatch. God forb...

Want to give your ears a break and read this as an article? You’re looking for this link.Never miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of February 15, 2021 with Corey Quinn.