[FLINK-10122] KafkaConsumer should use partitionable state over union state if partition discovery is not active

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: Connectors / Kafka
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned pull-request-available
• Assignee: Unassigned
• Reporter: Stefan Richter
• Created: 10/Aug/18 11:45
• Updated: 27/Dec/21 22:38

Description

KafkaConsumer store its offsets state always as union state. I think this is only required in the case that partition discovery is active. For jobs with a very high parallelism, the union state can lead to prohibitively expensive deployments. For example, a job with 2000 source and a total of 10MB checkpointed union state offsets state would have to ship ~ 2000 x 10MB = 20GB of state. With partitionable state, it would have to ship ~10MB.

For now, I would suggest to go back to partitionable state in case that partition discovery is not active. In the long run, I have some ideas for more efficient partitioning schemes that would also work for active discovery.

Issue Links

• links to: GitHub Pull Request #6537