[HUDI-3758] Optimize flink partition table with BucketIndex

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.1
• Component/s: flink
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: konwu
• Created: 31/Mar/22 07:35
• Updated: 18/Jun/22 19:56
• Resolved: 18/Jun/22 19:56

Description

When using flink bucket index , I meet two problems

• without use all streamWriter tasks when partition table with small Bucket number
• crashed with the following step

1. start job
2. killed before first commit success ( left some log files)
3. restart job run nomal after one successful commit
4. kill job and restart  throws `Duplicate fileID 00000001-6f57-4c71-bf6f-ee7616ec7b14 from bucket 1 of partition  found during the BucketStreamWriteFunction index bootstrap`

Issue Links

• links to: GitHub Pull Request #5185