[FLINK-11501] Add a ratelimiting feature to the FlinkKafkaConsumer

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 1.8.0
• Component/s: Connectors / Kafka
• Labels: pull-request-available
• Assignee: Lakshmi Rao
• Reporter: Lakshmi Rao
• Created: 31/Jan/19 23:22
• Updated: 10/Jul/19 01:55
• Resolved: 02/Mar/19 03:39

Description

There are instances when a Flink job that reads from Kafka can read at a significantly high throughput (particularly while processing a backlog) and degrade the underlying Kafka cluster.

While Kafka quotas are perhaps the best way to enforce this ratelimiting, there are cases where such a setup is not available or easily enabled. In such a scenario, ratelimiting on the FlinkKafkaConsumer is useful feature. The approach is essentially involves using Guava's RateLimiter to ratelimit the bytes read from Kafka (in the KafkaConsumerThread)

More discussion here: https://lists.apache.org/thread.html/8140b759ba83f33a22d809887fd2d711f5ffe7069c888eb9b1142272@%3Cdev.flink.apache.org%3E 

Issue Links

• duplicates: FLINK-5445 flink kafka consumer should support rate control
• relates to: FLINK-11826 Kafka09ITCase.testRateLimitedConsumer fails on Travis
• links to: GitHub Pull Request #7679
• links to: GitHub Pull Request #8081