[SPARK-43311] RocksDB state store provider memory management enhancements

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: Structured Streaming
• Assignee: Anish Shrigondekar
• Reporter: Anish Shrigondekar
• Created: 27/Apr/23 18:24
• Updated: 04/Jul/23 01:23
• Resolved: 01/May/23 18:51

Description

Today when RocksDB is used as a State Store provider, memory usage when writing using writeBatch is not capped. Also, a related issue is that the state store coordinator can create multiple RocksDB instances on a single node without enforcing a global limit on native memory usage. Due to these issues we could run into OOM issues and task failures. 
 
We are looking to improve this behavior by doing a series of improvements such as:

• remove writeBatch and use native RocksDB operations
• use writeBufferManager to manage global limit for all instances on a single node and accounting memtable + filter/index blocks usage as part of block cacheWith these changes we will be avoiding OOM issues around RocksDB native memory usage.