[SPARK-42819] Add support for setting max_write_buffer_number and write_buffer_size for RocksDB used in streaming

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: Structured Streaming
• Assignee: Anish Shrigondekar
• Reporter: Anish Shrigondekar
• Created: 16/Mar/23 01:00
• Updated: 16/Mar/23 10:52
• Resolved: 16/Mar/23 10:52

Description

Add support for setting max_write_buffer_number and write_buffer_size for RocksDB used in streaming

 

We need these settings in order to control memory tuning for RocksDB. We already expose settings for blockCache size. However, these 2 settings are missing. This change proposes to add them.

Issue Links

• links to: [Github] Pull Request #40455 (anishshri-db)
• links to: [Github] Pull Request #40455 (anishshri-db)