Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33114

Add metadata in MapStatus to support custom shuffle manager

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.1
    • Fix Version/s: None
    • Component/s: Shuffle
    • Labels:
      None

      Description

      Current MapStatus class is tightly bound with local (sort merge) shuffle which uses BlockManagerId to store the shuffle data location. It could not support other custom shuffle manager implementation.

      We could add "metadata" to MapStatus and allow different shuffle manager implementation to store information related to them.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              bobyangbo BoYang
              Shepherd:
              Felix Cheung
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: