[SPARK-38275] Consider to include WriteBatch's memory in the memory usage of RocksDB state store

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.1
• Fix Version/s: 3.3.0
• Component/s: Structured Streaming
• Assignee: Yun Tang
• Reporter: Yun Tang
• Created: 21/Feb/22 11:16
• Updated: 25/Feb/22 03:08
• Resolved: 25/Feb/22 03:08

Description

Current RocksDB state store actually use a unlimited WriteBatch with a DB, the WriteBatch would not be cleared until the micro-batch data committed, which results that the memoy usage of WriteBatch could be very huge.

We should consider to add the approximate memory usgae of WriteBatch as the totdal memory usage and also print it separately.

Issue Links

• links to: [Github] Pull Request #35600 (Myasuka)