[SPARK-40464] Support automatic data format conversion for shuffle state db

• Type: Improvement
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Yang Jie
• Created: 16/Sep/22 02:50
• Updated: 12/Oct/22 08:39

Description

After SPARK-38888,  Spark starts to support use RocksDB as the storage type of shuffle state db, but when users switch storage types, the original data store in `LevelDB/RocksDB` will not be automatically converted to another kind of storage now. The original data store will be retained and the new type data store will be  created when switching storage types.

This is not very friendly for migrating stock users to use RocksDB. If necessary, consider adding the function of automatic data format conversion:

• LevelDB -> RocksDB
• RocksDB -> LevelDB

 

 

Issue Links

• links to: [Github] Pull Request #38220 (panbingkun)