[HUDI-6962] Correct the behavior of bulk insert for NB-CC - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: New Feature
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.0.0-beta1
Component/s: None
Labels:
- pull-request-available

Epic Link:
Non-blocking concurrency control

Description

How to handle the case if the multiple writer contains a job with bulk insert operation?
1. Generated file group id: Generate a fixed file group ID because other jobs will use the fixed file group id suffix instead of random uuid suffix. The behavior needs to be consistent to prevent later writer jobs from writing the records with same primary key to different file groups.
2.Deal with the transaction: The conflict resolution of bulk insert could not defer to the compaction phase. Because bulk insert writers flush data into base files, if there are multiple bulk insert job, there might exists multiple base files in the same bucket.