[FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.8.0
• Component/s: Kafka Connector
• Assignee: Unassigned
• Reporter: Tzu-Li (Gordon) Tai
• Created: 13/Jan/17 10:49
• Updated: 18/Nov/18 12:41

Description

Reported in ML: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-topic-partition-skewness-causes-watermark-not-being-emitted-td11008.html

Similar to what's happening to idle sources blocking watermark progression in downstream operators (see FLINK-5017), the per-partition watermark mechanism in FlinkKafkaConsumer is also being blocked of progressing watermarks when a partition is idle. The watermark of idle partitions is always Long.MIN_VALUE, therefore the overall min watermark across all partitions of a consumer subtask will never proceed.

It's normally not a common case to have Kafka partitions not producing any data, but it'll probably be good to handle this as well. I think we should have a localized solution similar to FLINK-5017 for the per-partition watermarks in AbstractFetcher.

Issue Links

• is related to: FLINK-8001 Mark Kafka Consumer as idle if it doesn't have partitions
• links to: GitHub Pull Request #5634

Sub-Tasks

• Kafka consumer docs to warn about idle partitions: Closed
• Remove warning from kafka consumer docs: Open