Last Week In AWS Podcast

LinksThe Duckbill Group: https://www.duckbillgroup.com/TranscriptCorey: This episode is sponsored in part by Catchpoint. Look, 80 percent of performance and availability issues don’t occur within your application code in your data center itself. It occurs well outside those boundaries, so it’s difficult to understand what’s actually happening. What Catchpoint does is makes it easier for enterprises to detect, identify, and of course, validate how reachable their application is, and of course, how happy their users are. It helps you get visibility into reachability, availability, performance, reliability, and of course, absorbency, because we’ll throw that one in, too. And it’s used by a bunch of interesting companies you may have heard of, like, you know, Google, Verizon, Oracle—but don’t hold that against them—and many more. To learn more, visit www.catchpoint.com, and tell them Corey sent you; wait for the wince.Pete: Hello, and welcome to the AWS Morning Brief: Whiteboard Confessional. You are not confused. This is definitely not Corey Quinn. This is Pete Cheslock. I was the recurring guest. I've pushed Corey away, and just taken over his entire podcast. But don't worry, he'll be back soon enough. Until then, I'm joined by a very special guest, Jesse DeRose. Jesse, want to say hi?Jesse: Howdy everybody.Pete: Jesse and I are two of the cloud economists that work with Corey here at The Duckbill Group, and I convinced Jesse to come and join me today to talk about a new Amazon service that we had the pleasure—mm, you be the judge of that—of testing out recently, a service called Amazon Detective. This is a new service that I want to say was announced a couple of weeks ago, actually longer than that because, as you'll learn, it took us a little while to actually get a fully up and running version of this going, so we could actually do a full test on it. But as you can imagine, we get a chance to try out a lot of new Amazon services. And when we saw this service come out, we were pretty excited. Jesse, maybe you can chat a little bit about what piqued your interest when we first heard of Amazon Detective.Jesse: So, we here do a lot of analysis work with VPC Flow Logs. There's so much interesting data to be discovered in your VPC Flow Logs, and I really enjoy getting information out of those logs. But ultimately, digging into those logs via AWS’s existing services can be a bit frustrating; it can be a bit time-consuming in order to go through the administrative overhead to analyze those logs. So, for me, I was really excited about seeing how AWS Detective automatically allowed us to dig into some of that data, ideally more fluidly, or more organically, or naturally, to get at the same information with, ideally, less hassle.Pete: Exactly. So, for those that have not heard of AWS Detective yet, I'm just going to read off a little bit about what we read on the Amazon documentation that actually got us so excited. They talked a lot about these different security services like Amazon GuardDuty Macie, Security Hub, and all these partner products. But finding this central source for all of this data was challenging. And one of the things they actually called out which got us really excited is these few sentences. They said, “Amazon Detective can analyze trillions of events from multiple data sources such as Virtual Private Cloud (VPC) Flow Logs, AWS CloudTrail, and Amazon GuardDuty, and automatically creates a unified, interactive view of your resources, users, and the interactions between them over time.” It was actually this sentence that got us really excited because, as Jesse mentioned, we spend a lot of time trying to understand our clients’ data transfer usage. What is talking to what? Why is there charge for data transfer between certain services? Why is it so high? Why is it growing? And we spend, unfortunately, a lot of time digging around in the VPC Flow Logs. So, when we saw this, we got really excited because—well, Jesse, how do we do this today? How do we actually glean insight from Flow Logs?Jesse: It's a frustrating process. I feel like there has got to be a better way for us to get this information from a lot of our clients, and every single time we have to ask our clients to send over or share these VPC Flow Logs. There's that little wince of the implied. “I’m so sorry that we have to ask you to do it this way,” because it's doable, but it requires sinking data between S3 buckets, creating and running Athena queries, there's lots of little pieces that are required to build up to the actual analysis itself. There's no first-class citizens when it comes to analyzing these logs.Pete: It's really true. And Athena, the Data Factory—the Data Glue—what is it? Glue. You have to create a Glue Catalog. It's just a lot of work when we're really just trying to understand who and what are the top producers, consumers of data that is likely impacting spend for a client. So, we saw this and we thought to ourselves, “Wow, that one sentence it put in the list, it said, ‘The interactions between all of these resources and users over time.’” We got really excited for this. We also got excited because, of course, we love understanding how much things cost, but the pricing for Detective, it didn't seem that crazy. I mean, it's not great, but it's all based on ingested logs, which they don't really describe. So, our assumption is that if you send it your VPC Flow Logs, or CloudTrail logs, or whatever, you're going to pay for those on top of probably already paying for them today. So, that could be a deal-breaker for some clients out there.Jesse: That's the thing that was super frustrating for me, or super interesting for me is that AWS Detective, in terms of pricing and in terms of technology and capability, doesn't replace any of these other components. It is additive, which, generally speaking, I think is great, but when you start looking at it from a price perspective, that means that you're going to pay for CloudTrail logs, and VPC Flow Logs, and GuardDuty, and Macie, and all of these other services, and now you're going to pay for AWS Detective on top of that. So, it feels like you're paying twice for a lot of these services, when you could do a lot of the same analysis work yourself. And it's probably not going to be as clean to do it yourself in terms of building out the Glue Catalogs that we talked about building out, Athena tables and queries. But ultimately, it may be less expensive because it's not ultimately paying for all these additive services on top of each other.Pete: Exactly. I think we're definitely not being fair to the Amazon Detective product teams because we're trying to use this service, or we're hoping this service solves a really specific painful use case for us. And really, it's just based on what we found in their public-facing marketing.So, how does this actually work? Well, we found some really great information online via Amazon. They did a great job documenting how this all works. Essentially, you enable Amazon Detective, and you enable CloudTrail, and VPC, and GuardDuty, you have to enable it in multiple accounts, and Jesse can talk a little bit more about some of the caveats we ran into just setting it...

Links MentionedWant to give your ears a break and read this as an article? You’re looking for this link: https://www.lastweekinaws.com/blog/reader-mailbag-billing/SponsorsA Cloud Guru: https://acloudguru.comNew Relic: https://newrelic.comNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of September 21, 2020.

TranscriptCorey: This episode is sponsored in part by Catchpoint look, 80% of performance and availability issues don't occur within your application code in your data center itself. It occurs well outside those boundaries. So it's difficult to understand what's actually happening. What Catchpoint does is makes it easier for enterprises to detect, identify, and of course validate how reachable their application is. And of course, how happy their users are. It helps you get visible and to reach a bit availability, performance, reliability, of course, absorbency. Cause we'll throw that one in too. And it's used by a bunch of interns and companies you may have heard of like, you know, Google, Verizon, Oracle, but don't hold that against them. And many more. To learn more, visit www.catchpoint.com and tell them Cory sent you, wait for the wince.Welcome to the AWS Morning Brief: Whiteboard Confessional, now with recurring perpetual guest, Pete Cheslock. Pete, how are you?Pete: I'm back again.Corey: So, today I want to talk about something that really struck an awful lot of nerves across, well, the greater internet. You know, the mountains of thought leadership, otherwise known as Twitter. Specifically, Chef has gotten itself acquired.Pete: Yeah, I saw some, I guess you would call them, sub-tweets from some Chef employees before it was announced, which is kind of common, where responses ranged from, “Oh, that's something new,” to, “Welp.” And I've thought it—I was like, “Wow, that's interesting.” Of course, then I start looking for news of what happened, of which we all found out not long after.Corey: Before we go into it, let's set the stage here because it turns out not everyone went through the battles of configuration management circa 2012 to 2015 or so—at least in my experience. What did Chef do? What was the product that Chef offered? Who the heck are they?Pete: So, Chef, they were kind of a fast follower in the configuration management space to another very popular tool that I'm sure people have used out there called Puppet. Actually, interestingly enough, the founders of Chef ran a consulting company that was doing Puppet consulting; they were helping companies use Puppet. And both of those tools really came from yet another tool called CFEngine, which in many ways—depending on who you ask—it's kind of considered the original configuration management, the one that had probably the earliest, largest usage. But it was very difficult to use. CFEngine was not something that was easy, it had a really high barrier to entry, and tools like Puppet and Chef, they came out around the, let's say 2007, 8, 9 10 timeframe, were written in Ruby which was a little bit easier of a programming language to get up and running with. And this solved a problem for a lot of companies who needed to configure and manage lots of servers easily.Corey: And there are basically four companies in here that really nailed it for this era; you had Puppet, Chef, Salt, and Ansible. And in the interest of full disclosure, I was a very early developer behind SaltStack, and I was a traveling contract trainer for Puppet for a while. I never got deep into Chef myself for a variety of reasons. First and foremost was that its configuration language was fundamentally Ruby, and my approach back then—because I wasn't anything approaching a developer—was that if I need to learn a full-featured programming language at some point, well, why wouldn't I just pivot to becoming, instead, a developer in that language and not have to worry about infrastructure? Instead, go and build applications and then work nine to five and not get woken up in the middle of the night when something broke. That may have been the wrong direction, but that was where I sat at the time.Pete: Yeah, I came at it from a different world. So, I had worked for a startup that no one has probably really ever heard of, unless you have met me before, like, know who I am, but a company called Sonian which was very early in the cloud space. It was email archiving, so it wasn't anything particularly mind-blowingly interesting because it's compliant email archiving, but what was interesting is that we were really early in the cloud space, and a lot of the tools that people use today just didn't exist for managing cloud servers. It was 2008, 2009, pretty early, EC2 timeframe. How would you provision your EC2 instance, back then? Maybe you use CFEngine, maybe use Puppet. And actually, interestingly enough, that company—Sonian—was originally a Puppet shop because Chef didn't exist yet. And there were a series of issues we ran into, technical capabilities that Puppet just couldn't do for us at the time. And again, that time being 2009, 2010, and a lot of the very early Chef team, founding team, early engineers, were really working with us very closely to bootstrap our business on Chef writing a lot of those original cookbooks that became community cookbooks. And so, my intro into Chef and the Chef community is a lot earlier than most, and I went a lot deeper with it just by nature of being so early into that space.Corey: One of the things that struck me despite not being a Chef aficionado myself was, first, just how many people in the DevOps sphere were deeply tied into that entire ecosystem. And two, love or hate whatever the product, or company, et cetera, did, some of the most empathetic people I've ever met were closely tied to Chef’s orbit. So, I have not publicly commented until now on Chef getting acquired, just because I'm trying to give the people who are in that space, time to, I guess, I don't know if grieve is the right word, but it's important to me that I don't have a whole lot to say there, and it's very easy for me to say something that comes across as crass, or not well thought out, or unintentionally insulting to a lot of very good people. So, I'm sitting here on the sidelines watching it and more or less sitting this one out, but it's deeply affected enough people that I wanted to talk about it here.Pete: Yeah. And I'm glad that we are taking this opportunity to talk about it a bit. I had a lot of thoughts and feels. I tried to write a blog post about this to try to get them down somewhere, and a couple of paragraphs into it, I just, I really couldn’t… it just seemed like a meandering random mess of words without any real destination. But a few people online have mentioned this, and I'll definitely call it out as well, which is that Chef was, it was a tool. It was a tool like any other. You either loved it or you hated it. If you hated it, you probably really loved Ansible, or you really loved Puppet. It was a really, kind of, Vim versus Emacs feel to it, where you either we're all in on it or not.But the thing that I think Chef really brought for me is not only leveling up my career in a way that I would not be where I'm at today if it wasn't for that tool and that community, but just how genuine everyone was within that community, and the interactions that we had at conferences, at Chef conferences, DevOps conferences and things of that nature, and even continued the conversations online back before Slack, which it's hard to even remember that: when we all were on IRC, and we were in the Chef IRC channel, and it was a fantastic channel with a ton of people who would dive in and help you out on your Chef problems....

Links MentionedWant to give your ears a break and read this as an article? You’re looking for this link: https://www.lastweekinaws.com/blog/is-the-aws-free-tier-really-free/SponsorsA Cloud Guru: https://acloudguru.comNew Relic: https://newrelic.comNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of September 14th, 2020.

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.LinksTrend MicroChaosSearch@QuinnyPigTranscriptCorey: This episode is brought to you by Trend Micro Cloud One™. A security services platform for organizations building in the Cloud. I know you're thinking that that's a mouthful because it is, but what's easier to say? “I'm glad we have Trend Micro Cloud One™, a security services platform for organizations building in the Cloud,” or, “Hey, bad news. It's going to be a few more weeks. I kind of forgot about that security thing.” I thought so. Trend Micro Cloud One™ is an automated, flexible all-in-one solution that protects your workflows and containers with cloud-native security. Identify and resolve security issues earlier in the pipeline, and access your cloud environments sooner, with full visibility, so you can get back to what you do best, which is generally building great applications. Discover Trend Micro Cloud One™ a security services platform for organizations building in the Cloud. Whew. At trendmicro.com/screaming.Corey: Welcome to the AWS Morning Brief. I'm Cloud Economist Corey Quinn, And this is the Whiteboard Confessional segment that has increasingly been taken over by my colleague, Pete Cheslock, as we tear through various cloud-based companies, public filings as they race to go public and inflict their profits, or in more cases, losses on the public markets. Pete, thanks for joining me.Pete: I am super happy to be back again, and making my mother happy that I'm actually using that MBA that I spent all that time to get.Corey: So, we could wind up talking just about how Palantir is awful in a variety of ways. My personal favorite was the letter that their CEO attached saying that effectively engineers were stupid and didn't see the big picture, which is a weird thing to say about a whole group of people you're actively trying to hire, but all right. Let's talk about their S-1 filing. This has been anticipated for a while. What do you think?Pete: Well, Palantir has been around for a very long time. I think it's been around a lot longer than a lot of people realize. You know, early 2000s. It was technology built to tie data together and to be honest, I only know—I’ve ever heard of one company actually using Palantir—the technology—a commercial company. They were actually using it as a SIM—SIM, whatever you want to call it—Security Information Management System—Corey: Event management or something like that. Yeah.Pete: Exactly. And ironically enough, that company actually—that was using Palantir—replaced it with an Elasticsearch ELK stack, which I thought was fascinating. I know nothing about their software, but I was very fascinated to read the S-1 because there's been this mythology around it and you can hear so much about insiders at Palantir, employees selling their shares in this wide secondary market. So, I was very curious to see what we were going to find, and there are definitely some interesting bits within.Corey: There certainly are. And it's strange because for a while Palantir was doing interesting things in the market. They were offering $20,000 referral bonuses to people who referred engineers in for certain roles, and you didn't have to be a Palantir employee to do it, which was fascinating. They've recently moved headquarters from Palo Alto over to Denver, Colorado, which… okay. They are claiming it's for this whole lofty mission. Let's not kid ourselves: it's a tax play. [laughs]. And there's also a whole bunch of interesting stuff buried in here. But yeah, in many ways, this is a legacy company in some respects. It's been around almost 20 years. And strangely, I don't know about you, but I don't know anyone who works for Palantir. I did a little digging in preparation for this episode, and it turns out, I actually kind of do, but they're very quiet about it. It's one of those things where people don't want to be called out for working at a company that is this particular flavor of controversy, and I can't say I blame them.Pete: Yeah, I haven't looked through my LinkedIn to see if any of my connections have ever worked there. Granted, it's such a West Coast company that me out in the East Coast, be pretty rare to run into anyone out here who's kind of taken their time and done the Palantir. I have heard, again, the rumors that they've always paid very well, and—Corey: They would kind of have to.Pete: You know, in the Bay Area, you kind of have to. And competing for talent against other places who pay really well, like Netflix, and Uber, and all these other big companies that are out there. So, it's a big competition for the top talent.Corey: Oh, yeah. And most of what they do is data analytics. They take in a whole bunch of data, and they crunch a whole bunch of numbers and come out with other stuff. Historically, they have been focused on selling their services to governments, but now they're expanding in the enterprise story as well. And that is, of course, going to be a bit of a challenge for them as they expand into it, but we can talk about what they do, how they do it, and all the other challenges. Let's talk about Cloud. What do we know about their cloud environment based upon their public filing?Pete: Well, they talk about their commitments. So, this is something you often see in S-1s of their various cloud commitments, and I think this one was super interesting in that they listed commitments for about $1.5 billion in cloud commitments over six years, and this was an agreement they entered into at the end of last year. Just a massive, massive amount of cloud spend commitment, right?Corey: Yeah, it’s a quarter billion dollars a year in spend. Which is, again, we see a number of customers in that range pretty frequently, it's not always typical to see the better part of a decade done to satisfy those commitments, though. Usually they're, “Well, this stuff is always changing. Let's talk about doing this for the next three years.” Six is a bit on the outside range of what we tend to see. What's fun to me was the breakdown of that commitment, which was just—I've been using this as a talking point for a week now—which is they have to undisclosed cloud companies in this part. They mention elsewhere that they use Azure and that they use AWS. Great. Fine. For one cloud provider, they have a six-year commitment of $1.49 ...

Links MentionedWant to give your ears a break and read this as an article? You’re looking for this link: https://www.lastweekinaws.com/blog/dipping-my-toes-into-the-digitalocean SponsorsA Cloud Guru: https://acloudguru.comNew Relic: https://newrelic.comNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of September 7, 2020.

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.LinksTrend Micro Cloud One™ChaosSearchTranscriptCorey: This episode is brought to you by Trend Micro Cloud One™. A security services platform for organizations building in the Cloud. I know you're thinking that that's a mouthful because it is, but what's easier to say? “I'm glad we have Trend Micro Cloud One™, a security services platform for organizations building in the Cloud,” or, “Hey, bad news. It's going to be a few more weeks. I kind of forgot about that security thing.” I thought so. Trend Micro Cloud One™ is an automated, flexible all-in-one solution that protects your workflows and containers with cloud-native security. Identify and resolve security issues earlier in the pipeline, and access your cloud environments sooner, with full visibility, so you can get back to what you do best, which is generally building great applications. Discover Trend Micro Cloud One™ a security services platform for organizations building in the Cloud. Whew. At trendmicro.com/screaming.Corey: Welcome to the AWS Morning Brief: Whiteboard Confessional series, where I am joined once again by my colleague, Pete Cheslock. Pete, thanks for taking the time to tolerate my slings, arrows, and other various forms of cynicism.Pete: You know, I didn't take that much offense to the fact that I have an MBA, so I decided to come back and see if we can make use of that investment.Corey: So, fun story. The last one of these that we did was talking about—who's one was that? They're starting to run together at this point.Pete: That was Sumo Logic’s.Corey: That's right. And it was, “Oh, let’s talk about what they're doing.” And then throughout the day, I think five tech companies all filed to go public, which is just bizarre. So, we're going to take a couple more episodes to slice and dice a couple more that were of interest to us.Pete: Yeah, absolutely. We're going to chat about one that I was honestly been waiting for because of the hype and the myths around this company. But it's a big data company called Snowflake.Corey: They're very special and unique.Pete: They're very special. I think—I often will listen to CNBC in the background, it's kind of interesting to get little words, and sometimes tech pops up into a CNBC broadcast. When Snowflake filed. I think one of the announcers had said something to the effect of, “I don't know why you'd want to be called Snowflake.” [laughs]. So, I had a good chuckle at that one.Corey: Because they've been around longer then that's been a disparaging term used by jerks.Pete: [laughs]. Exactly, exactly. So, they filed their S-1 in that flurry with a whole slew of other companies—which we will definitely get to at least one more of those—and honestly, this company, I've never worked there. I did at one point go through a sales process which I can share some of my thoughts and opinions there, but the reason why I was so excited to see this one is because of the sheer amount of VC money that this company has raised, well over—I don't know if ‘well over’ but definitely over a billion dollars of VC funding raised. It's crazy.Corey: My comment at one of the big tech conferences last year—back when that was a thing we went to—was I was walking around their booth, and I noticed that they had this mock-up of a race car suspended in the air. And then I realized, “Oh, my God, that isn't a mock-up.” Which told me at that point that if you're paying retail pricing for Snowflake, you're probably doing something very wrong.Pete: Yeah, absolutely. I think to dive into one of my favorite Snowflake stories, at a previous company, we were checking Snowflake out—we got connected with them via some connections our head of product had and some success that we heard that Snowflake had with helping, you know, a data warehouse. That's what it is: it's a data warehouse technology. If you're in the Amazon ecosystem, you might be using Redshift. Snowflake can do some of those things, it can do some other things.Corey: Why would I use something like Snowflake instead of Redshift? I mean, for starters, naive approach as well, okay, this is in a different Amazon account, so at minimum, I'm going to be paying data transfer in and out on both sides. But again, we're talking data warehousing, the data transfer is usually something of a rounding error compared to all the extra cost goes into that.Pete: And this is where I think a lot of their growth in the early days came from a lot of the deficiencies in Redshift. In technologies, in the investment that Amazon was doing there, Snowflake could do a lot of things just simply better. I think additionally, too, they were probably taking a lot of business from Oracle shops and things of that nature. But I do know a friend of mine at his company, they had a well over a million dollars a month in Redshift spend, and they actually moved over to Snowflake as a cost-savings initiative. It was significantly cheaper. But what’s, I think, so fascinating, when I heard that I was like, “Well, hold on a second. You know, Snowflake runs inside of Amazon.” So, I'm always curious of how that relationship exists with Amazon where you've got some account manager who's going to lose on some big spend of an Amazon customer by their Redshift spend going down dramatically, but then whoever the account manager for Snowflake must just be super excited by that because obviously their spend is going to go up.Corey: Yeah, on some level, if you're running a data warehouse on top of AWS, from the high-level AWS perspective, well, is it spend that’s going to happen on your account, or is it spend that’s going to happen on Snowflakes account? It's not likely that you're going to be building everything on top of AWS, and then Snowflake is going to be running its stuff on another provider. The data transfer charges there become exceedingly non-trivial.Pete: Yeah, absolutely. One of the things that is interesting about how Snowflake works, at least from my recollection a few years ago, is that you can stream your data into S3 which is very cost-effective. Snowflake can actually ingest your data from S3, and what they basically do is they put it into their S3. And you pay the same S3 pricing. I remember the sales guy. He w...