[HUDI-3187] Initiate RFC Lock free concurrency control

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Duplicate
• Fix Version/s: 1.0.0-beta1
• Component/s: writer-core
• Epic Link: Multi-writer & OCC
• Assignee: Vinoth Chandar
• Reporter: Vinoth Chandar
• Created: 06/Jan/22 18:58
• Updated: 05/Jul/23 18:38
• Resolved: 05/Jul/23 18:38

Description

Basic idea here builds on principles laid out here. 

https://cwiki.apache.org/confluence/display/HUDI/RFC+-+22+%3A+Snapshot+Isolation+using+Optimistic+Concurrency+Control+for+multi-writers#RFC22:SnapshotIsolationusingOptimisticConcurrencyControlformultiwriters-FutureWork(LockFree-ishConcurrencyControl) 

 

At a high level, we want to introduce tracking of "transition_time" to the Hudi timeline, where we track the following for each action. 

• start_time (current instant time)
• action type
• state (requested, inflight, completed)
• transition_time (when the action reached that given state) 

Then, we could use this to resolve conflicts during read/query planning time. 

Goal of this task, to write up a detailed design doc around this and kickstart this. 

We already have a WIP PR that establishes some feasibility and lays some ground work