[SPARK-50163] Fix Streaming RocksDB AcquireLock double release

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.3
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Livia Zhu
• Reporter: Livia Zhu
• Created: 29/Oct/24 21:51
• Updated: 02/Nov/24 22:40
• Resolved: 02/Nov/24 22:40

Description

The RocksDB implementation will run into a race condition with the acquireLock due to the completion listener in the following scenario: 

Task 1	Task 2
load() - acquire
commit() - release
	load() - acquire()
completionListener - release

Now RocksDB is thread unsafe as task 2 believes it holds the lock though the lock was released. 

Issue Links

• links to: GitHub Pull Request #48697