[HUDI-4065] Implement a self-releasing FileLock for multiple writers scenario

• Type: New Feature
• Status: Closed
• Priority: Blocker
• Resolution: Done
• Fix Version/s: 0.12.0
• Component/s: multi-writer
• Labels: pull-request-available
• Epic Link: Multi-writer & OCC
• Assignee: Jian Feng
• Reporter: Jian Feng
• Created: 09/May/22 13:16
• Updated: 04/Jan/23 01:11
• Resolved: 26/Jul/22 13:54

Description

when using HiveMetastoreBasedLockProvider or ZookeeperBasedLockProvider, if the writer with locker crash before releasing the locker,  no writers can obtain the locker anymore, hence can not cleaning/expiring the lock after a failed write.

Instead of those lockers depending on external components, FIleBaseLocker only relies on FileSystem's atomic file creation behavior.  an instant will be written to the file, so it will clean the lock when failing writer rollback. also can set expire strategy on it.

 

Issue Links

• links to: GitHub Pull Request #6071
• links to: GitHub Pull Request #6138