[HUDI-845] Allow parallel writing to Hudi tables using Locking service

• Type: New Feature
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: writer-core
• Labels: help-requested pull-request-available
• Assignee: Nishith Agarwal
• Reporter: Vinoth Chandar
• Created: 29/Apr/20 05:00
• Updated: 30/Sep/21 06:00
• Resolved: 23/Mar/21 14:00

Description

Things to think about 

• Commit time has to be unique across writers 
• Parallel writers can finish commits out of order i.e c2 commits before c1.
• MOR log blocks fence uncommited data.. 
• Cleaner should loudly complain if it cannot finish cleaning up partial writes.  
• Move the pending rollback work into cleaner

 

P.S: think about what is left for the general thing : log files may have different order, inserts may violate uniqueness constraint

Issue Links

• Blocked: HUDI-2503 HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service
• is a child of: HUDI-1456 [UMBRELLA] Concurrency Control for Hudi writers and table services
• links to: GitHub Pull Request #2374

Sub-Tasks

• Remove pendingRollbacks and move rollbacks into cleaner/archival path: Open