[GOBBLIN-1922] Create work units for selected topics partitions in Kafka source

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: gobblin-kafka
• Assignee: Shirshanka Das
• Reporter: Hanghang Liu
• Created: 03/Oct/23 18:25
• Updated: 26/Oct/23 17:13

Description

Add a new feature in Kafka source to create workUnit for selected topics partitions. This feature can provide functionality for:

1. Recompute and split tasks(workUnit) during runtime for selected topics partitiions, instead of recompute for all Kafka topics
2. Make topic level replan feasible
3. One step closer to dynamic work unit allocation 

Need to have a followup to make work unit packer able to pack the recomputed workunits into a desired number of containers