[SPARK-38178] Correct the logic to measure the memory usage of RocksDB

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.1
• Fix Version/s: 3.3.0, 3.2.2
• Component/s: Structured Streaming
• Assignee: Yun Tang
• Reporter: Yun Tang
• Created: 10/Feb/22 15:20
• Updated: 11/Feb/22 03:38
• Resolved: 11/Feb/22 03:38

Description

Currently, structed streaming would report the memory usage of RocksDB as state store's metrics. However, it only sums from rocksdb.estimate-table-readers-mem and rocksdb.size-all-mem-tables. However, this lacks of the memory usage of block cache. (You can refer to [Memory-usage-in-RocksDB](https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB) for more details.)

BTW, as the "block-cache-pinned-usage" is included in "block-cache-usage", we don't need to include the pinned usage.

Issue Links

• links to: [Github] Pull Request #35480 (Myasuka)
• links to: [Github] Pull Request #35480 (Myasuka)