Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov

Tim Berglund is joined by Viktor Gamov (Developer Advocate, Confluent) to discuss various approaches to testing Kafka Streams applications.EPISODE LINKSKafkaEmbeddedTopologyTestDriverMocked Streams (Scala)MockafkaTest containersKafka containers SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

It’s a problem endemic to the tech world that we are always focused on what’s coming next, that we often forget to look at where we’ve been. Chris Riccomini, who was there at LinkedIn when Apache Kafka® was born, tells us how Kafka and the stream processing framework Samza came about, and also what he’s doing these days at WePay—building systems that use Kafka as a primary datastore.EPISODE LINKSWhen It Absolutely, Positively, Has to be There: Reliability Guarantees in KafkaSo, You Want to Build a Kafka Connector? Source Edition.Kafka is Your Escape HatchSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Gwen and Kai chat about machine learning architectures, and whether software engineers and data scientists can learn to get along.EPISODE LINKSBlogs on deploying machine learning workloads: Machine Learning with Python, Jupyter, KSQL and TensorFlowHow to Build and Deploy Scalable Machine Learning in Production with Apache KafkaUsing Apache Kafka to Drive Cutting-Edge Machine LearningKIP-392: Allow consumers to fetch from closest replica Watch the video version of this podcastSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

It has been said that in distributed messaging, there are two hard problems: 2) exactly once delivery, 1) guaranteed order of messages and 2) exactly once delivery. Apache Kafka® has offered exactly once processing since version 0.11, which allows properly configured producers and consumers to make the guarantee that each message will be processed exactly one time. In this episode, Kafka Streams engineer Guozhang Wang walks through the implementation of transactional messaging in Kafka in some detail, including the idempotent producer API, the transaction coordinator responsible for managing the transaction log and consumer configurations. It’s a complex topic, but he takes us through it carefully and completely.EPISODE LINKSTransactions in Apache Kafka Enabling Exactly Once in Kafka Streams KIP-98: Exactly Once Delivery and Transactional MessagingKIP-129: Streams Exactly-Once SemanticsSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Stanislav Kozlovski joins us to discuss common pitfalls when using Kafka consumers and a new KIP that promises to make consumer restarts much smoother.EPISODE LINKSKIP-345: Static consumer membership KIP-211: Documents the current behavior of offset expirationWatch the video version of this podcastSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Microservices are pretty ubiquitous these days. Really “SOA done right,” they reimagine the services pattern in the context of the world we live in today, nearly two decades since the first big service-oriented systems hit production. But what have we learned in this time? There are plenty of war stories. System designers have explored different architectural patterns—REST, events and databases of all types. In this podcast, Tim Berglund and Ben Stopford explore the event-driven paradigm and how it relates to the microservice architectures we build today. Ben dives deep into coupling, evolution and challenges of our increasingly data-oriented culture. He also talks about the future, where data are events and events are data, and touches on real-time architectures that retain the decoupling properties needed to be pluggable, and to evolve. Powerful stuff.EPISODE LINKSDesigning Event-Driven Systems Building a Microservices Ecosystem with Kafka Streams and KSQLSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

After several years of development, librdkafka has finally reached 1.0! It remains API compatible with older versions of the library, so you won’t need to make any changes to your application. There are, however, several important new features like the idempotent producer, sparse broker connections, support for the vaunted KIP-62 and a complete makeover for the C#/.NET client.EPISODE LINKSlibrdkafka v1.0.0 release notesSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Do metrics for detecting clients from old versions actually exist? Or is Gwen making features up? This and more useful advice is coming up on today's episode of Ask Confluent.EPISODE LINKSThe Java property that will refresh DNS cache frequently: java.security.Security.setProperty(“networkaddress.cache.ttl” , “60");Improvements to DNS lookups in Confluent Platform 5.1.2 (Apache Kafka 2.1.1):KAFKA-7755KAFKA-7890More reasons to upgrade to Confluent Platform 5.1.2Monitoring clients with old versions:KIP-188 has lots of important new metrics If you are worried about “down-conversion” as discussed in Ask Confluent #5, you want to monitor: MBean: kafka.server:type=BrokerTopicMetrics,name=FetchMessageConversionsPerSec,topic=([-.\w]+)KIP-188 also added a metric for temp memory usage (memory used for conversion and compression) that can be usefulIn KIP-272, we’ve added version tag to request metrics, so you can see how many requests per sec you get from each versionRecommendations for Kafka Summit NYCWatch the video version of this podcastSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Nick Dearden explains the five stages of streaming maturity. They are not denial, anger, bargaining, depression and acceptance—that’s the Kübler-Ross model, and it’s for bad things. This one is for awesome things, and takes you from the first streaming project you ever build all the way to a state where an entire organization is transformed to think in terms of real-time, event-driven systems. If you have ever found yourself trying to get streaming technology adopted, this episode is for you!EPISODE LINKSFive Stages to Streaming Platform AdoptionSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Kubernetes provides all the building blocks needed to run stateful workloads, but creating a truly enterprise-grade Apache Kafka® platform that can be used in production is not always intuitive. In this episode, Tim Berglund and Viktor Gamov address some of the challenges and pitfalls of managing Kafka on Kubernetes at scale. They also share lessons learned from the development of the Confluent Operator for Kubernetes, and answer questions like:-What is Kubernetes?-What are stateful workloads?-Why are they hard?-Will Confluent Operator make it easier?EPISODE LINKSJoin the #kubernetes Slack channelKafka on Kubernetes: Does it really have to be “The Hard Way”?SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.