[SPARK-45503] RocksDB State Store to Use LZ4 Compression

• Type: Task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.5.1
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Siying Dong
• Reporter: Siying Dong
• Created: 11/Oct/23 22:47
• Updated: 24/Oct/23 20:25
• Resolved: 24/Oct/23 20:25

Description

LZ4 is generally faster than Snappy. That's probably we use LZ4 in changelogs and other places by default. However, we don't change RocksDB's default of Snappy compression style. The RocksDB Team recommend LZ4 or ZSTD and the default is kept to Snappy only for backward compatible reason. We should use LZ4 instead.

Issue Links

• links to: GitHub Pull Request #43338