Kafka on Azure Brief

Apache Kafka is an open-source distributed event streaming platform used for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. It is a highly distributed publish-subscribe messaging system with a robust queue that can handle a high volume of data from one endpoint to another. Kafka messages are persistent on the disk and can be replicated across multiple nodes in a cluster to prevent data loss. Kafka offers high Reliability, Scalability, Durability and Performance, with an emphasis on zero down time and zero data loss.

Kafka is used often for data monitoring involving aggregating statistical data from distributed applications, or as a log aggregation solution, where data is collected from multiple services and then parsed in a standard format for consumption. Kafka is also popular with frameworks like Storm and Spark Streaming where streaming data from a topic is collected, processed, and then written as a new topic, where it becomes available for consumers or applications to consume.

Figure: Source Confluent Docs

For our tests, we used the performance tests packaged with the Kafka release builds. We used the producer performance tests in the bin directory, under the name kafka-producer-perf-test.sh.

We have focused on two metrics – latency and throughput.

The latency test was run with a single producer client against a 3-node Kafka cluster while measuring the end-to-end response time of the requests.

For the throughput test, two D64psV5 Instances were used. We simultaneously ran two producers on each instance, and throughput was calculated by aggregating the individual throughputs as reported by each producer.

For our testing, we chose to use 4 vCPU instances – the E4psv5 VMs – each equipped with a 32 TB Premium SSD LRS disk for storing the Kafka Data. All instances, including the load generators and the Kafka clusters, were running Ubuntu 20.04 OS (Operating System). The Kafka version used is 2.13, and OpenJDK-11-jdk was used. We used a 100-byte record size with all the workloads. A batch.size of 65,536 was chosen and linger.ms was set to 5 based on internal testing. These values could change depending on the developer’s requirements. For data durability we have set the acks value to 1.

As can be seen in the chart above, we observed that the Latency of a single producer workload of Apache Kafka on the Microsoft Azure E4psV5 instance, powered by the Ampere Altra processor, is comparable with the AMD Milan-based EsV5 instance and 50% better than the Intel Icelake-based EsV5 instance. Lower latencies lead to reduced time for content to be made available to downstream consumers.

As can be seen in the chart above, the aggregated throughput of the Kafka Cluster hosted on the E4psV5 instance, powered by Ampere Computing, has a 25% cost advantage over the E4sV5 instance. The Cost benefit represented here shows only the benefit from the Instance type using the Azure Instance Pricing Calculator.

Kafka is a popular distributed event store and stream-processing platform that is in use by thousands of companies today. It aims to provide a unified, high throughput, low latency platform for handling real-time data feeds. As can be seen in this workload brief, Azure EpsV5 virtual machines, powered by the Ampere Altra Cloud Native processors can yield up to a third lower p.99 latencies while making no compromises on performance, leading to overall better price-performance, all while reducing your carbon footprint.

Ampere Altra processors are designed to deliver exceptional performance for cloud native applications like Redis. They do so by using an innovative architectural design, operating at consistent frequencies, and using single-threaded cores that make applications more resistant to noisy neighbor issues. This allows workloads to run in a predictable manner with minimal variance under increasing loads. The processors are also designed to deliver exceptional energy efficiency. This translates to industry leading performance/watt capabilities and a smaller carbon footprint.