Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27412

Add a new shuffle manager to use Persistent Memory as shuffle and spilling storage

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: Shuffle, Spark Core
    • Labels:
    • Flags:
      Patch

      Description

      Add a new shuffle manager called "PmemShuffleManager", by using which, we can use Persistent Memory Device as storage for shuffle and external sorter spilling.

      In this implementation, we leveraged Persistent Memory Development Kit(PMDK) to support transaction write with high performance.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                xuechendi Chendi.Xue
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: