Last Week In AWS Podcast

Want to give your ears a break and read this as an article? You’re looking for this link.SponsorsNew RelicExtraHop Never miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of January 18th, 2021 with Corey Quinn.

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.TranscriptCorey: When you think about feature flags—and you should—you should also be thinking of LaunchDarkly. LaunchDarkly is a feature management platform that lets all your teams safely deliver and control software through feature flags. By separating code deployments from feature releases at massive scale—and small scale, too—LaunchDarkly enables you to innovate faster, increase developer happiness—which is more important than you’d think—and drive transformation throughout your organization. LaunchDarkly enables teams to modernize faster. Awesome companies have used them, large, small, and everything in between. Take a look at launchdarkly.com, and tell them that I sent you. My thanks again for their sponsorship of this episode.Pete: Hello, and welcome to the AWS Morning Brief: Friday From the Field. Triple F; that's what we're calling it now. We’re going a new direction. I'm Pete Cheslock.Jesse: I'm Jesse DeRose, and I'm so excited for Triple F.Pete: Triple F. Hashtag Triple F. So, moving away, taking this into a new direction, we have… not stolen that's a little bit too aggressive. But we have been lovingly gifted this podcast from Corey Quinn after taking over while he was on paternity leave, we just kept on doing it; we never stopped, we never let him have it back. And he was nice enough just to give us this opportunity to take this Friday podcast into a new direction and talk about things that we're seeing as cloud economists in the field working with our clients.Jesse: Yeah, it really started as this confessional discussion of weird architecture patterns that we've seen, but then it definitely morphed into more of the other things that we've seen from either our work with Duckbill or work with previous engagements or previous companies. So, it just felt fitting to rebrand just ever so slightly and focus more of our efforts on what are the things that we're seeing day-to-day? What are the major problems that our clients are seeing? What are some of the pain points we've seen? What are the new features from AWS that are really the interesting and important things to talk about?Pete: Exactly. We have an interesting insight that I think a lot of folks in the industry don't get to see. We, for one, look at countless Amazon bills, seeing how people are spending their money. But we also are often reached out to directly to help engineering teams better answer questions that they're getting from finance. I mean, that's the biggest fear I have—Jesse: Yeah.Pete: —CFO comes walking over to my desk, and I haven't submitted an expense report recently like, what do they want?Jesse: [laugh]. I didn't do it. It wasn't me.Pete: Even worse is when some of your executives start learning some of these terms. And they say, “Hey, what's our cost per unit on Amazon Cloud?”Jesse: Yeah, it is something that has morphed from just a conversation about engineering teams thinking about their architecture patterns and what might be best for them to getting the entire company involved—especially finance—to ask all these questions and really think about, what's the bottom line here? How can we better understand this cloud spend?Pete: I know most people are probably thinking, “Doesn't tagging solve this problem. Can’t I just tag everything, and then I have all my answers, right?” Problem solved.Jesse: I'm sorry, did you just tell me to go F myself there, Pete?Pete: [laugh]. Obviously, we both know that even the best of companies, the most mature companies we work with, yeah, they might be about 90% plus fully tagged, but even those companies still have to put in a lot of effort to answer these questions and to understand where their spend is going. Because they say, that which gets measured gets improved. So, are you measuring your spend? Are you measuring your growth? Do you understand how your spend changes as usage changes, your customers change? I mean, there's countless questions. But there's another thing that we see, too, Jesse, right? This circle of pain, the—what is it—the cost management circle of pain.Jesse: Yeah. Yeah. It's this really fascinating idea focusing on cloud cost optimization, where a company will realize that their cloud spend has gone up for whatever reasons, and they say, “Oh, no. We need to do something about this.” Whether that is because finance has come over and asked the question, or because engineering has caught the issue. And so they go through this quick session, maybe a quarter, maybe a couple months or more of figuring out, “How can we cut costs? Can we remove resources? Can we put these practices into place? Can we build some processes? Okay, now, everything's fine, right? We've managed to bring our costs back down. We managed to get rid of all of those EBS snapshots that were collecting dust and never to be used, so now we can go about business as usual again, right?” And so then they continue on as if nothing has happened. And without making long term changes, those costs are going to rise again. And then all of a sudden, we're back in the same spot of, “Oh, no, our cloud costs have gone up, why did they go up? We did all these things to make sure that we didn't have run into this issue again. Why are our cloud costs going up again?” And the cycle just repeats. It's a really unfortunate kind of spiral.Pete: I remember my time at a startup where we were under a series of really high growth, a lot of customers coming on the platform. And my favorite meeting ever was the CEO talking about our financials. And he mentioned that our gross margin was negative 175%, which for the non-financial folks, means that for every dollar of income negative 175% is being spent for that. You normally want that number to be positive if you want to have a successful business. And remember, the line he said is, “We are going to successfully go out of business with a gross margin that is negative one hundred and seventy”—whatever I said. This is an important number that people need to think about. And what's amazing is that within a year, we had turned that around to be an extremely high gross margin because we started looking, and tracking, and bringing cultural change, and giving ownership to people to own these numbers. So, it's not just an engineering problem anymore. Everyone thinks that the Amazon bill is because your engineers built a certain thing, or turned on a certain type of instance. And sure, part of that is absolutely true, but I always like to say that your Amazo...

Special thanks to Alice Goldfuss for this week’s awesome title!Want to give your ears a break and read this as an article? You’re looking for this link.SponsorsVeeamNew RelicNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of January 11, 2021 with Corey Quinn.

TranscriptCorey: Software powers the world. LaunchDarkly is a feature management platform that empowers all teams to safely deliver and control software through feature flags. By separating code deployments from feature releases at scale, LaunchDarkly enables you to innovate faster, increase developer happiness, and drive DevOps transformation. To stay competitive, teams must adopt modern software engineering practices. LaunchDarkly enables teams to modernize faster, Intuit, GoPro, IBM, Atlassian, and thousands of other organizations rely on LaunchDarkly to pursue modern development and continuously deliver value. Visit us at launchdarkly.com to learn more.Pete: Hello, and welcome to the AWS Morning Brief. I’m Pete Cheslock.Jesse: I'm Jesse DeRose.Pete: And we're back yet again. We're well into 2021. I mean, about a week or so, right?Jesse: I'm excited. I'm just glad that when midnight struck. I didn't roll back over into January 1st of 2020.Pete: Yeah, luckily, it's not a Y2K scenario. I don't think we have to deal with the whole date issues until, what, 2032 I think, whatever that the next big Y2K-ish date issue is going to be. I'm hopefully retired by the time that that happens. Jesse: That's future us problem. Pete: Yeah. Future us problem, absolutely. Well, we've made it. We've made it to 2021, which is a statement no one thought they were going to say last year at this point.Jesse: [laugh].Pete: But here we are. And today, we're talking about an interesting topic that may bring us some hate mail. I don't know. You tell me, folks that are listening. But we're seeing this more and more in our capacity as cloud economists working with clients here at The Duckbill Group, that folks who are running Kubernetes—whether it's EKS, or they're running it on EC2 using maybe, like, an OpenShift—are actually spending more than people who are using other primitives within AWS. So, we wanted to chat a little bit about why we think that is, and some of the challenges that we're seeing out there. And we would love to hear from you on this one. If you are using Kubernetes in any of the ways that we're going to talk about, you can actually send us a story about how you're doing that and maybe answer some of these questions we have, or explain how you're using it. If you go to lastweekinaws.com/QA to ask us questions—not quality assurance—but go to QA for asking us questions. You can put in your information, you can add your name, it's optional if you want. You can be completely anonymous and just tell us how much you enjoy our wonderful tones and talking about technology. So, Kubernetes. Why is this the thing, Jessie?Jesse: I feel like when it first came out, it was the hot thing. Like, everybody wanted Kubernetes, everybody wanted to be Kubernetes, there were classes on Kubernetes, there were books on—like, I feel like that's still happening. I think it has amazing potential in a lot of ways, but I also feel like… in the same way that you might read the Google SRE book and then immediately turn to your startup team of three people and say, “We're going to do everything the way that Google does it,” this isn't always the right option.Pete: Feel like the Google SRE book is, like, The Mythical Man Month, which is, the book that everyone wants to quote, the name of the book, but none of those people have ever actually read the book.Jesse: Yeah, there's lots of really great ideas, but just because they're great ideas that worked well for a large company at scale doesn't necessarily mean that they're going to be the same right ideas for your company.Pete: And also, we're both fairly grizzled former system administrators and operators; Kubernetes is not the first, kind of, swing of the bat at this problem. I mean, we've had Mesos which, it's still around but not as hip and cool; we've had OpenStack. Does—remember when all the Kubernetes people were all like, “Nope, OpenStack is going to be the greatest thing ever.” So, needless to say, we are a little jaded on the topic.Jesse: You can't forget about Nomad, either, from HashiCorp built cleanly into HashiCorp’s Hashi stack with all of their other amazing development and deployment tools. Pete: Yeah. I mean, this is a problem that people want to solve. But in the rise of Cloud, on Amazon I always struggled with why it was needed. And we're going to talk a little bit about that. So, again, what is Kubernetes? I hope people are listening that would know this, but maybe not. It's an abstraction layer for scheduling workloads. It's the solution to the Docker problem. Like, a container is great. I have a container, it is a totally self-contained application, ready to go, my configuration, my dependencies. And now I need a place to run it. Well, where do I run this container? Well, pre-Kubernetes, Jessie, you'd probably use something like ECS—the Elastic Container Service—might be a way that you could schedule some workloads. Jesse: Or maybe if you just wanted to run a single virtual machine somewhere and run that container in the virtual machine, you might do that as well. Pete: Yeah, that was how a lot of the earliest users of Docker were just running Docker: they were just running the containers as applications—because that's what they are—on their bare EC2. They would just run some EC2 and run a Docker container on there. And there were benefits to that. You got this isolated package deployed out there not having to worry about dependencies. You have to worry about having the right Python dependencies or Ruby dependencies. It came with everything it needed, and that was a big solution. Now Kubernetes, I think, brings this really interesting concept that I like. It's this API that theoretically you could use in a lot of different places. If you now have this API to deploy your application anywhere there's a Kubernetes cluster, does this solve vendor-lock-in? Could you use Kubernetes to solve some of these issues that we see?Jesse: You could use Kubernetes to solve vendor-lock-in in the same way that you could use multi-cloud to solve vendor lock-in. Again, it is a solution to the problem, but is it the right solution for your company?Pete: That is always the question I feel like I would ask folks when they were using Kubernetes is, I would always ask why they were using it. I honestly will say I never got—I don’t want to say wouldn't say never; that's not fair. I rarely would get a good answer. It was often like a little bit of operational FOMO—you know, the fear of missing out on the next hottest thing, which of course, that's never a good way to pick your architecture stack. Now, that being said, at a previous company, we were investigating Kubernetes to solve a problem with our stateless applications—because I in no way trusted it to run anything stateful. None of my databases I wanted on it. But it is a great way to put more control into my developers’ hands-on deploying their applications. We ran predominantly C class instances on EC2. And th...

Want to give your ears a break and read this as an article? You’re looking for this link.SponsorsVeeamExtraHopNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of January 4, 2021 with Corey Quinn.

Links#AWSWishList@AWSWishList AccountFollow Pete + Jesse on TwitterTranscriptCorey: When you think about feature flags (and you should), you should also be thinking of LaunchDarkly. LaunchDarkly is a feature management platform that lets all your teams safely deliver and control software through feature flags by separating code deployments from feature releases at massive scale (and small-scale too), LaunchDarkly enables you to innovate faster, increase developer, happiness (which is more important than you think), and drive transformation throughout your organization. LaunchDarkly enables teams to modernize faster. Awesome companies have used them, large, small, and everything in between. Take a look at launchdarkly.com to learn more and tell them that I sent you. My thanks again for their sponsorship of this episode.Pete: Hello and welcome to the AWS Morning Brief. I am Pete Cheslock.Jesse: I'm Jesse DeRose.Pete: We are welcomed yet again with Amy Negrette.Amy: Hello.Pete: We are here. We made it. It is actually 2021.Jesse: I can tell you flying cars: definitely a thing. World peace: we're close, we're so close.Pete: We're so close. Well, guess what? We made it, we survived 2020. And with it, we brought with us part two of the #awswishlist. So, this is where we went through—especially as leading up to re:Invent and getting through re:Invent—we went through and looked at the Twitter hashtag of #awswishlist so that we could pick out some of our favorite things, some #awswishlist items that we think are important to us, or just interesting in their own right. We'll include the link to these tweets in the [00:01:57 show notes]. So definitely go check that out, and you can check out the conversation, or maybe follow some of that to see when things actually come around. But yeah, we'll just walk through some of the things we found that were pretty interesting and chat about why we hope Amazon includes them into a future release. So, one thing that I saw which I thought was pretty interesting because I run into this problem also, is a way of downloading data from various third party locations directly into S3, Dynamo, or some sort of data store location. Essentially, it'd be awesome to just completely get rid of having services around, or Fargates, or Lambdas set up for downloading data from places that—how cool would it be? And this is, again, not an enterprise-y type feature, but just, like, a personal thing of how cool would it be to be, like, I want to take this ISO from a place and just put a URL in S3 and say, “Put that thing in this thing,” and call it a day. So, again, a personal complaint of mine plus, also, someone else tweeted it, so there's two people out there that want this—at least—so therefore Amazon, you got to build it for me.Amy: Those are the rules.Pete: Those are the rules. Right. Right, Amy, those are the rules. Jesse: And I feel like, let's be honest, that ISO that you want to download anyway is probably living in S3 somewhere else anyhow. So, it's just moving bucket to bucket.Pete: Someone has that, you know, Slackware ISO that I've been looking for, from, you know, 2001. It's in someone else's bucket; just let me have it myself. Exactly. Amy, what did you find in your discovery of the #awswishlist hashtag?Amy: This is a thing that I think really should be on any of these on-demand pay-as-you-go services because AWS really targets those [00:03:48 unintelligible] markets for a lot of their serverless deployments. And this actually came from one of my friends who had this problem on Twitter, where you need to be able to set a maximum on on-demand spend, let's say in his case, Dynamo. So, you don't hypothetically build in a loop and spend a whole bunch of money. Pete: Yeah.Amy: And really, it should be in anything that does that. If it's not telling you something where I'm only wanting to run this much because it's on-demand, then you should be able to control that spend somehow.Pete: And with the—what is it—millisecond billion on Lambda, you can get really granular bills for your poorly architected Lambda functions. Jesse: I feel like computers are the best because they'll do exactly what you want them to do, except for when they do what you tell them to do and not what you actually want them to do, and that drives me absolutely insane. So, I'm with you. I think that this is a great opportunity.Amy: That problem will be solved when the robots take over.Pete: [laugh]. One of my favorite discoveries of doing our kind of Duckbill cost optimizations where we dive into people's spend and help them architect things new was finding a Lambda function that was taking longer and longer to execute—meaning, costing more money—by putting more and more data into a poorly configured Dynamo table that was also causing it to take longer and longer. And so not only did you have a Dynamo table that was poorly configured, taking this data and taking longer to do it, you were just getting a hit on both sides. It happens.Jesse: That hurts my soul. Pete: So, what’d you find, Jesse? What was some of the good wishlist items that you're hoping for in 2021?Jesse: So, I come from a background of a lot of infrastructure as code I've worked a lot with Terraform, I know enough about Chef to be dangerous to your production environment. One thing that I saw a couple people tweet about that I would love to see is mock AWS API endpoints for, effectively, unit tests for a lot of infrastructure as code. Because if you think about when you're building infrastructure as code, the only way that you can really test it is by running it, by actually seeing, “Can I actually create the resources that I think I'm creating with this infrastructure as code content?” So, I would love to see maybe a feature flag for AWS services through the API where you can say, “Hey, don't actually create this RDS database or this EC2 instance, but just return the results as if I did create it. Maybe leave the Instance ID blank or something like that.” And then you, in writing your unit tests, can confirm all the details that you would expect to see in that response. Pete: I feel like there was a—Atlassian, maybe, had a project that was something like this, some sort of a way of unit testing these things. Again, it was something on GitHub, so even if it was associated with a large publicly traded enterprise, I'm sure it's fallen into disrepair at this stage.Jesse: [laugh]. I will say I found an open-source tool looking into this one, called LocalStack that allows you to basically spin up an instance on your local machine that acts as the AWS API endpoint so that it actually creates this mock endpoint for you locally on your machine. But effectively, I'd love to see th...

Want to give your ears a break and read this as an article? You’re looking for this link.SponsorsExtraHopLinodeNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill