Every day, our data ingest pipeline handles petabytes of malicious internet traffic events. This high-volume pipeline heavily relies on Kafka, with hundreds of producers and consumers running on Kubernetes pods and performing thousands of read and write operations per second. Consequently, Kafka rebalance has become a familiar challenge we face. Join us for this session focused on valuable insights into mitigating the disruptions caused by rebalance storms.
During this session, you'll discover effective techniques for optimizing consumer group configurations, minimizing unnecessary rebalances, and ensuring uninterrupted data processing. Gain practical strategies to maintain pipeline stability and enhance the overall performance of your Kafka-driven pipelines, even in the face of dynamic scaling and restarts.
Yulia Antonovsky is a Senior Software Engineer II at Akamai
With more than 15 years of experience in the tech industry, Yulia has been focusing on Data Engineering for the past 6 years.
She holds a BSc in Computer and Information Sciences from the Technion (Israel Institute of Technology).