Every day, our data ingest pipeline handles petabytes of malicious internet traffic events. This high-volume pipeline heavily relies on Kafka, with hundreds of producers and consumers running on Kubernetes pods and performing thousands of read and write operations per second. Consequently, Kafka rebalance has become a familiar challenge we face. Join us for this session focused on valuable insights into mitigating the disruptions caused by rebalance storms.

During this session, you'll discover effective techniques for optimizing consumer group configurations, minimizing unnecessary rebalances, and ensuring uninterrupted data processing. Gain practical strategies to maintain pipeline stability and enhance the overall performance of your Kafka-driven pipelines, even in the face of dynamic scaling and restarts.