Last Week In AWS Podcast

AWS Morning Brief for the week of December 28, 2020 with Corey Quinn.

Links#AWSWishList@AWSWishList AccountFollow Pete + Jesse on TwitterTranscriptCorey: This episode is sponsored in part by our friends at Linode. You might be familiar with Linode; they’ve been around for almost 20 years. They offer Cloud in a way that makes sense rather than a way that is actively ridiculous by trying to throw everything at a wall and see what sticks. Their pricing winds up being a lot more transparent—not to mention lower—their performance kicks the crap out of most other things in this space, and—my personal favorite—whenever you call them for support, you’ll get a human who’s empowered to fix whatever it is that’s giving you trouble. Visit linode.com/screaminginthecloud to learn more, and get $100 in credit to kick the tires. That’s linode.com/screaminginthecloud.Pete: Hello and welcome to AWS Morning Brief. I am Pete Cheslock. I'm joined yet again with Jesse DeRose. We are also excited to re-invite recurring guest for number two, Amy Negrette. Say hello, Amy.Amy: Hello.Pete: So, we are here. This is Christmas. Or should I say Christmahanukwanza. Jesse: So, close. That works. Pete: So, close. But it's the Christmahanukwanza episode—Hanu—hanukwanza—Jesse: Christmashanukwanzika. Pete: And if you thought Hanukkah was spelled a bunch of different ways, Christmahanukwanza is spelled a lot of different ways. And we are here to talk about the #amazonwishlist, which is honestly one of my favorite hashtags to follow on Twitter—#awswishlist. It is pretty popular, it's heavily used.Jesse: It was actually so heavily used that they made a specific @awswishlist account, basically, specifically to follow a lot of these hashtags, and to re-highlight a lot of these hashtags, especially when some of the wishes are actually fulfilled. Pete: Yeah, I think it's a great thing, and if I was an Amazon product manager, I would love this too because just talk about making my job a lot easier, I guess. Jesse: One thing that I do want to call out, I was looking through a number of the tweets going around for the hashtag#awswishlist, and I noticed that there was some of the responses from AWS folks, which one I'd love to say thank you, AWS for actually taking this seriously and actually responding to folks in conversation on Twitter for these wishlist items. There was one that I found where the person directed the original poster to an AWS support page, which was basically AWS’s, like, ‘Contact Us’ page. And the Contact Us page basically said, “Hey, if you have some questions, here's what you should do. I have some questions that could help improve an AWS product or service, how can I send feedback to AWS?” And all the answers were, “Click the feedback button on the page that you're on, either in the AWS console or the AWS documentation, or contact AWS support directly.” So, close—Pete: Did you just tell me to go F myself there, Jesse? [laugh].Jesse: [laugh]. I didn't maybe say it in so many words, but I think I did.Amy: I absolutely love it when a support page says, “Maybe you should just do it yourself.” And I'm like, “Well if I did, I probably wouldn't have been here in the first place.”Pete: Exactly. So, what we decided to do, what we thought would be kind of fun, is to troll through the Twitter #awswishlist hashtag and take a look at what people were saying, especially because it's a lot busier around the pre to current re:Invent time. And so independently each of us put together a list of things that—I mean, at least I could speak for myself—I thought were interesting, or things that I thought would be cool to have. And yeah, we're just going to talk about them and see from there. So, we'll include a link to each of these tweets in the [00:04:18 show notes] so you can check them out, and also so you can see the conversation on them. What was also cool, I just want to call out is that some of these that we saw on there, at least that I saw have been resolved by re:Invent time. One was AWS CloudShell that was announced recently at re:Invent, someone was saying I want is this AWS CloudShell thing because other vendors have this: Azure has this, Google has this. So, here's a scenario where Amazon was catching up. So, I thought that was pretty cool to see. So, I'm going to kick it off because, whatever, I'm here, and I got my list in front of me. So, this is actually related to the CloudShell one, which I thought was interesting. So, there was some conversation online about CloudShell, and this is maybe potentially allowing people to remove the need of having a bastion host, which, how cool is that you don't have to run those anymore? Jesse: Oh, yeah. Pete: And so there was a question around, “Well, does my identity get a home directory?” Which sounds like the answer was “Yes.” But the question mark there had to do when using AWS SSO because it has to do with the IAM principle, it's what comes back from the sts get-caller-identity. So, if you are using one of the different Federation technologies, your actual identity could be different for each one. And so that's a wishlist item that I could definitely be on board with because if you're dealing with IAM roles or Federation, and your home directory is never the same, that can be kind of annoying. Jesse: I cannot tell you how many times I have downloaded a file or put a file somewhere on a bastion host, gone away to a different project, come back to it, or SSH’ed into the same bastion host and wondered why it wasn't there anymore, only to realize that I was on a different bastion host in a different environment, or that the data had been purged every so often for security or cleaning purposes. I would absolutely love clean roles and just really, really well defined boundaries on this. Coming from somebody who uses different AWS accounts on a regular basis for the different clients that we work with, I would just love to see this really kind of clean structure of AWS, IAM usage, and user management and security.Pete: And, Jesse, we saw similar issues, I believe, when we were playing around with QuickSight, and Federation, and IAM so—Jesse: Oh, yes.Pete: Hopefully that gets a little bit fixed up. But anyway, I thought that was a pretty interesting one. Amy, what did you find in your discovery of the Amazon wishlist hashtag?Amy: I did find one for X-Ray support in API Gateway HTTP API. Again, one of the worst, longest names of any service, and EventBridge, which surprisingly, one that this hasn't happened yet, but two, [00:07:12 unintelligible] for me is kind of a double-edged sword where it's one of those services that everyone needs, but als...

Want to give your ears a break and read this as an article? You’re looking for this link.SponsorsExtraHopLinodeNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of December 21, 2020 with Corey Quinn.

LinksFollow Pete + Jesse on TwitterTranscriptCorey: This episode is sponsored in part by our friends at Linode. You might be familiar with Linode; they’ve been around for almost 20 years. They offer Cloud in a way that makes sense rather than a way that is actively ridiculous by trying to throw everything at a wall and see what sticks. Their pricing winds up being a lot more transparent—not to mention lower—their performance kicks the crap out of most other things in this space, and—my personal favorite—whenever you call them for support, you’ll get a human who’s empowered to fix whatever it is that’s giving you trouble. Visit linode.com/screaminginthecloud to learn more, and get $100 in credit to kick the tires. That’s linode.com/screaminginthecloud.Pete: Hello, and welcome to AWS Morning Brief. I am Pete Cheslock.Jesse: And I'm Jesse DeRose.Pete: We're back again, and we're here to answer an audience question. So, every once in a while people tweet at us—you can tweet me @petecheslock. Jesse, what is your Twitter handle?Jesse: @Jessie_DeRose.Pete: Yeah, mine is just petecheslock. I do feel bad for the other Pete Cheslock, who actually does live in Boston as well because taking all of his profile names.Jesse: You should change yours to @therealpetecheslock, or he should change his to @therealpetecheslock, and then it'll just be an ongoing escalating battle.Pete: That's very true. So, occasionally on the Twitters, we get questions asked of whatever around Amazon cost management, things like that. And we wanted to actually take this opportunity to answer one of the more interesting questions that we received. Because granted, sometimes we get questions and they're pretty boring, so we don't answer them. We just focus on the fun ones, [laugh]—Jesse: [laugh].Pete: —selfishly, but we got this question that was really interesting. It had to do with someone who is essentially starting over within Amazon Web Services, meaning they were going to be redeploying their application into a series of new AWS accounts. And they asked us, “What are the most recent best practices—” I hate that term, but the important things you should do and consider when you're deploying into Amazon, into AWS. And we kind of sat back, we thought to ourselves, “Wow, how often does someone have that opportunity?” Right, Jesse?Jesse: Yeah. Not in any of my experience has that happened for me. I'm very, very envious of these people.Pete: Yeah, I had that opportunity one time, where we were essentially doing that, like, net-new, starting over. But this was years ago, where there wasn't a lot of insight into this, and we didn't have the features like we have today where Amazon organizations—AWS Organizations—allows such an easy way to create accounts and get started with multiple accounts. So, anyway, we want to take this opportunity to talk about what we believe and what we see as the things that you should focus on, what you should optimize for when getting started, when creating, kind of, net-new in AWS.Jesse: Yeah, there's a lot of different things that you can optimize for in AWS, and it really depends on what your business goals are; what do you ultimately want to accomplish when you are deploying your application into the cloud? But one of the big ones that we see, selfishly, here at Duckbill Group is cost optimization. And so we wanted to talk a little bit more about cost allocation and cost attribution—which are essentially the same thing, we may use the terms interchangeably in this conversation—to talk about how you can think about cost attribution, why you should think about cost attribution and some of the best ways to go about implementing that in AWS as you're building these new accounts, this new space.Pete: Yeah, and that being said, I really like people to really think when they create these things. Again, what are you optimizing for? Some people might say, “Oh, well, we want to optimize for security.” And that's great. You absolutely should do that.Jesse: Sure.Pete: Security is a first principle, something to absolutely focus on. But what if I told you that the other, probably, most important thing in AWS is—and something if you're not doing it today, you're going to be asked to do it in the future—is accurate cost attribution. And what if you could do both highly secure accounts, and segment based on security, but also get this cost attribution? That is, I think, what we're going to dive into today.Jesse: Yeah, I think that there's a lot of big conversations around engineers, and multiple other teams when you start talking about the DevOps movements, the DevSecOps movements, all these movements of the software engineers who are actually writing the code and the engineers or the operations folks who are—maybe—managing the infrastructure, maybe deploying the code, maybe the software engineers are deploying the code, it really depends on your team setup. But there's this, kind of, idea that the engineering teams that are working with this code, and then there's all these other teams in the company that have other things that are their top priority, and starting to bridge that gap to have conversations with finance to better understand what do they need to know from you about how you're spending money in AWS, and security who wants to better understand are we patched for the upcoming audit? Are we compliant based on these terms? It's really important to start thinking about how you optimize in AWS based on those ideas, those conversations with other teams. So, that's kind of ultimately what I'm thinking about, specifically, today, specifically about the conversation between finance and engineering and talking about cost attribution.Pete: But Jesse, aren't tags supposed to solve all of my problems when it comes to cost allocation?Jesse: [laugh]. Oh, I wish. They are supposed to. There's that whole idea of ‘set it and forget it,’ there's a big movement of ‘tag it and forget it,’ and as much as I want to believe in that, it’s unfortunately just not true. Like, tagging is definitely a first step, but it goes so much further than tagging and I think that's one of the big things that a lot of folks miss or don't think about when they're talking about tagging and cost attribution.Pete: If you loved it, you would have put a tag on it.Jesse: [laugh].Pete: But really, while tagging is an important thing to do, and we've seen some of our clients, their tagging percentages can be upwards of 90 percent, which is herculean in ability and effort to reach that level of coverage, but even then, getting that last 5 to 10 percent in many cases could be actually impossible to do because there can be a series of spend within Amazon which is just untaggable, or at least untaggable in a realistic way. And that's where multiple accounts can really help your busine...

Want to give your ears a break and read this as an article? You’re looking for this link.SponsorsVeeamNewRelicNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of December 14, 2020 with Corey Quinn.

LinksFollow Last Week In AWS on TwitterAWS Outage Message"Kinesis Outage" by Ryan FrantzTranscriptCorey: This episode is sponsored in part by our friends at Linode. You might be familiar with Linode; they’ve been around for almost 20 years. They offer Cloud in a way that makes sense rather than a way that is actively ridiculous by trying to throw everything at a wall and see what sticks. Their pricing winds up being a lot more transparent—not to mention lower—their performance kicks the crap out of most other things in this space, and—my personal favorite—whenever you call them for support, you’ll get a human who’s empowered to fix whatever it is that’s giving you trouble. Visit linode.com/screaminginthecloud to learn more, and get $100 in credit to kick the tires. That’s linode.com/screaminginthecloud.Pete: Hello, everyone. Welcome to the AWS Morning Brief. It's Pete Cheslock again—Jesse: And Jesse DeRose.Pete: We are back to talk about ‘The Kinesis Outage.’Jesse: [singing] bom bom bum.Pete: So, at this point, as you're listening to this, it's been a couple of weeks since the Kinesis outage has happened, and I'm sure there are many, many armchair sysadmins out there speculating at all the reasons why Amazon should not have had this outage. And guess what? You have two more system administrators here to armchair quarterback this as well.Jesse: We are happy to discuss what happened, why it happened. I will try to put on my best announcer voice, but I think I normally fall more into the golf announcer voice than the football announcer voice, so I'm not really sure if that's going to play as well into our story here.Pete: It's going, it's going, it's gone.Jesse: It’s—and it's just down. It's down—Pete: It's just—Jesse: —and it's gone.Pete: No, but seriously, we're not critiquing it. That is not the purpose of this talk today. We're not critiquing the outage because you should never critique other people's outages; never throw shade at another person's outage. That's not only crazy to do because you have no context into their world. It's just, it's not nice either, so just try to be nice out there.Jesse: Yeah, nobody wants to get critiqued when their company has an outage and when they're under pressure to fix something. So, we're not here to do that. We don't want to point any fingers. We're not blaming anyone. We just want to talk about what happened because honestly, it's a fascinating, complex conversation.Pete: It is so fascinating and honestly, loved the detail, a far cry from the early years of Amazon outages that were just, “We had a small percentage of instances have some issues.” This was very detailed. This gave out a lot of information. And the other thing too is that, when it comes to critiquing outages, you have to imagine that there are unlikely to be more than a handful of people even inside Amazon Web Services that fully understand the scope of the size and the interactions of all these different services. There may not even be a single person who truly understands how these dozens of services interact with each other. I mean, it takes teams and teams of people working together to build these things and to have these understandings. So, that being said, let's dive in. So, the Wednesday before Thanksgiving, Kinesis decided to take off early. You know, long weekend coming up, right? But really, what happened was is that there was an addition of capacity to Kinesis, and it caused it to hit an operating system limit causing an outage. But interestingly enough—and what we'll talk about today—are the interesting and downstream effects that occurred via CloudWatch, Cognito, even the status page, and the Personal Health Dashboard. I mean, that's a really interesting contributing factor or a correlating outage. I don't know the words here, but it's interesting to hear that both CloudWatch goes down and the Personal Health Dashboard goes down.Jesse: That's when somebody from the product side says, “Oh, that's a feature, definitely not a bug.”Pete: But the outage to CloudWatch then even affected some of the downstream services to CloudWatch—such as Lambda—which also included auto-scaling events. It even included EventBridge, which was impacted, and that even caused some ECS and EKS delays with provisioning new clusters and scaling of existing clusters.Jesse: So, right out of the bat, I just want to say huge kudos to AWS for dogfooding all of their services within AWS itself: not just providing the services to its customers, but actually using Kinesis internally for other things like CloudWatch and Cognito. They called that out in the write-up and said, “Kinesis is leveraged for CloudWatch, and Cognito, and for other things, for various different use cases.” That's fantastic. That's definitely what you want from your service provider.Pete: Yeah, I mean, it's a little amazing to hear, and also a little terrifying, that all of these services are built based on all of these other services. So, again, the complexity of the dependencies is pretty dramatic. But at the end of the day, it's still software underneath it; it's still humans. And I don't want to say that I am happy that Amazon had this outage at all, but watching a company of this stature, of this operational expertise, have an outage, it's kind of like watching the Masters when Tiger Woods duffs one into the water or something like that. It's just—it's a good reminder that—listen, we're all human, we're all working under largely the same constraints, and this stuff happens to everyone; no one is immune.Jesse: And I think it's also a really great opportunity—after the write-up is released—to see how the Masters go about doing what they do. Because everybody at some point is going to have to troubleshoot some kind of technology problem, and we get to see firsthand from this, how they go about troubleshooting these technology problems.Pete: Exactly. So, of course, one of the first things that I saw everywhere is everyone is, on mass, moving off of Amazon, right? They had an outage, so we're just going to turn off all our servers and just move over to GCP, or Azure, right? Jesse: Because GCP is a hundred percent uptime. Azure is a hundred percent uptime. They're never going to have any kind of outages like this. Google would never do something to maybe turn off a service, or sunset something.Pete: Yeah, exactly. So, with the whole talk about hybrid-cloud and multi-cloud strategies, you got to know that there's a whole slew of people out there, probably some executive at some business, who says, “Well, we need to engineer for this type of durability, this type of thing to happen again,” but could you even imagine the complexity...

Want to give your ears a break and read this as an article? You’re looking for this link.SponsorsVeeamLinodeNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of December 7, 2020 with Corey Quinn.