[FLINK-22839] Assign-evenly kafkaTopicPartitions of multiple topics to flinkKafkaConsumer subtask

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.11.0
• Component/s: Connectors / Kafka
• Assignee: Unassigned
• Reporter: Xu xiaolong
• Created: 02/Jun/21 02:36
• Updated: 08/Jun/21 09:24
• Resolved: 08/Jun/21 09:24

Description

Now ,with flink1.11 kafka connecotr,when we consume multiple kafka topics by one flinkkafkaconsumer , when we set the consumer parallelism equals with the total partitions count of multiple topic , make a decision each topic partition consume by one kafka consumer, so each topic partition count is less than the subtask count. But,the problem is that currently, some subtask is total free while someothers workload is very high, this problem is caused by that the partitionAssigner assign partion of earch topic indepently currently.

Following is one example: Target topics: topi1, topic2 ,topic3 ,topic4.  each has 3 partitions. In our job we consume the 4 topic by one consumer , our flink standalone cluster got 9 taskworkers on different nodes. we want balance the workload as much as possible, so we  set the paralelism of flinkkafkaconsumer to 12. from the UI we notice that the 0-5 subtask is free without partition assigned, the total 12 partiton is assigned to 6-11 subtask. We learned the source code of KafkaTopicPartitionAssigner to explain this phenomenon ,and then we extend one more partition assign strategy which can deal with the need we describe up, this stategy can evenly assign partiton from multiple topic grobally to subtask of consumer. we want to contibute to flink, so someone has the same requirement can use it directlly.

Issue Links

• duplicates: FLINK-22840 Assign-evenly kafkaTopicPartitions of multiple topics to flinkKafkaConsumer subtask