Major Description
Today when we call KafkaConsumer.poll(), it will fetch data from Kafka asynchronously and is put in to a local buffer (completedFetches).
If now we pause some TopicPartitions and call KafkaConsumer.poll(), we might throw away any buffered data that we might have in the local buffer for these TopicPartitions. Generally, if an application is calling pause on some TopicPartitions, it is likely to resume those TopicPartitions in near future, which would require KafkaConsumer to re-issue a fetch for the same data that it had buffered earlier for these TopicPartitions. This is a wasted effort from the application's point of view.
At Linkedin, we made a hotfix to see if NOT throwing away the prefetched data would improve the performance for stream applications like Samza. We ran a benchmark to compare the "before-fix" and "after-fix" versions.
We had a consumer subscribed to 10 partitions of a high volume topic and paused predefined number partitions for every poll call. The partitions to pause were chosen randomly for each poll() call.
- Time to run Benchmark = 60 seconds.
- MaxPollRecords = 1
- Number of TopicPartition subscribed = 10
| Number Of Partitions Paused | Number of Records consumed (Before fix) | Number of Records consumed (After fix) |
|---|---|---|
| 9 | 2087 | 4884693 |
Updated June 24, 2019
I followed up with mgharat on the status of this work since the current patch PR is stale. This work would also be beneficial to the Alpakka Kafka connector, which frequently pauses partitions as a means of back-pressure from upstream Akka Streams graph stages. I've reviewed the PR feedback from hachikuji and reimplemented this solution to add completed fetches that belong to paused partitions back to the queue. I also rebased against the latest trunk which caused more changes as a result of subscription event handlers being removed from the fetcher class.
I created a sample project that simulates the pause partition scenario that mgharat described above. It only uses the Kafka client instead of a stream processor like Samza or Alpakka Kafka. Even without setting max.poll.records to 1 there are significant gains in the number of records consumed and the amount of traffic between the consumer and brokers. I created two versions of the sample project, one based on the latest available Kafka Client Consumer (2.2.1) and one based on the new patch (2.4.0-SNAPSHOT). Each app has its own topic with its own producers and is constrained with cgroups. For full details of the experiment see the [K8s resources in this branch.
I exported a Grafana snapshot for public viewing. I included a screenshot in the attachments.
