[SPARK-45419] Avoid reusing rocksdb sst files in a dfferent rocksdb instance by removing file version map entry of larger versions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0, 3.5.1
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Chaoqin Li
• Reporter: Chaoqin Li
• Created: 05/Oct/23 00:38
• Updated: 10/Oct/23 02:05
• Resolved: 10/Oct/23 02:03

Description

When loading a rocksdb instance, remove file version map entry of larger versions to avoid rocksdb sst file unique id mismatch exception. The SST files in larger versions can't be reused even if they have the same size and name because they belong to another rocksdb instance.

Issue Links

• links to: GitHub Pull Request #43174