Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25299 Use remote storage for persisting shuffle data
  3. SPARK-31801

Register shuffle map output metadata with a shuffle output tracker

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Shuffle
    • None

    Description

      Part of the design as discussed inĀ this document.

      Establish a ShuffleOutputTracker API that resides on the driver, and handle accepting map output metadata returned by the map output writers and send them to the output tracker module accordingly.

      Requires https://issues.apache.org/jira/browse/SPARK-31798.

      Attachments

        Activity

          People

            Unassigned Unassigned
            mcheah Matt Cheah
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: