[SPARK-27412] Add a new shuffle manager to use Persistent Memory as shuffle and spilling storage

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 3.0.0
• Component/s: Shuffle, Spark Core
• Labels: core
• Flags: Patch
• Assignee: Unassigned
• Reporter: Chendi.Xue
• Created: 09/Apr/19 06:45
• Updated: 10/Apr/19 13:34
• Resolved: 10/Apr/19 13:34

Description

Add a new shuffle manager called "PmemShuffleManager", by using which, we can use Persistent Memory Device as storage for shuffle and external sorter spilling.

In this implementation, we leveraged Persistent Memory Development Kit(PMDK) to support transaction write with high performance.

 

Issue Links

• links to: GitHub Pull Request #24322