[HUDI-7740] Sharing locks across distributed Hudi writers

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Fix Version/s: 1.0.0
• Component/s: metadata
• Epic Link: STS stability
• Assignee: Ethan Guo (this is the old account; please use "yihua")
• Reporter: Lin Liu

Reviewers

Danny Chen, Jonathan Vexler, Sagar Sumit, Vinoth Chandar

• Created: 09/May/24 19:56
• Updated: 09/May/24 20:00

Description

One Hudi table can be ingested by multiple writers concurrently. Without proper synchronization, data corruptions can happen easily, and the most straight forward method is to use locks. In-process locks can be used for writers in the same JVM. For distributed writers, a distributed lock should be utilized.

Given a distributed lock has been generated for these writers, our immediate question is to share the lock across all these writers reliably. In this effort, we aim to utilize the `.hoodie` folder as the central place to share lock information. 

The goal of this effort is to make sure in this concurrent scenario,

1. Each write operation should be guarded with a lock.
2. All writers utilize the same lock at any moment.
3. When there is an lock update, the lock update operation should satisfy the above two conditions.