[SPARK-48742] Virtual Column Family for RocksDB

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Epic Link: Structured Streaming - Arbitrary State API v2
• Assignee: Jing Zhan
• Reporter: Jing Zhan
• Created: 27/Jun/24 21:35
• Updated: 09/Jul/24 03:02
• Resolved: 09/Jul/24 03:02

Description

Introducing virtual column family to RocksDB. We attach an 8-byte-Id prefix as column family identifier for each of the key row that is put into RocksDB. The encoding and decoding of the virtual column family prefix happens at the RocksDBKeyEncoder layer as we can pre-allocate extra 8 bytes and avoid additional memcpy.

Issue Links

• links to: GitHub Pull Request #47107