[SPARK-34198] Add RocksDB StateStore implementation

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Component/s: Structured Streaming
• Target Version/s: 3.2.0
• Assignee: Yuanjian Li
• Reporter: L. C. Hsieh
• Shepherd: L. C. Hsieh
• Created: 22/Jan/21 00:43
• Updated: 16/Dec/21 01:41
• Resolved: 09/Oct/21 16:24

Description

Currently Spark SS only has one built-in StateStore implementation HDFSBackedStateStore. Actually it uses in-memory map to store state rows. As there are more and more streaming applications, some of them requires to use large state in stateful operations such as streaming aggregation and join.

Several other major streaming frameworks already use RocksDB for state management. So it is proven to be good choice for large state usage. But Spark SS still lacks of a built-in state store for the requirement.

We would like to explore the possibility to add RocksDB-based StateStore into Spark SS.

 

Issue Links

• relates to: SPARK-28120 RocksDB state storage
• links to: [DESIGN] RocksDB State Store for Structured Streaming

Sub-Tasks

• Move general StateStore tests to StateStoreSuiteBase: Resolved
• The implementation of RocksDBCheckpointMetadata: Resolved
• RocksDBFileManager - save checkpoint to DFS: Resolved
• RocksDBFileManager - load checkpoint from DFS: Resolved
• Implementation for RocksDB instance: Resolved
• Cleanup support for RocksDB instance: Resolved
• Metrics support for RocksDB instance: Resolved
• The implementation for RocksDBStateStoreProvider: Resolved
• Introduce the RocksDBStateStoreProvider in the programming guide: Resolved
• RocksDB state store: Add additional metrics for better observability into state store operations: Resolved
• Upgrade the RocksDB version to 6.20.3: Resolved
• Store the RocksDB format in the checkpoint for a streaming query: Resolved