[GOBBLIN-1455] Limit gobblin.kafka.minContainersForTopic config to the number of partitions of the topic

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.15.0
• Fix Version/s: 0.17.0
• Component/s: gobblin-kafka
• Assignee: Shirshanka Das
• Reporter: Sudarshan Vasudevan
• Created: 01/Jun/21 17:52
• Updated: 14/Jun/23 04:13

Description

We need to upper bound the config gobblin.kafka.minContainersForTopic so as not to exceed the number of partitions for the topic. When this config value exceeds the number of partitions, current behavior incorrectly estimates the work unit size in KafkaTopicGroupingWorkUnitPacker.