Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25299 Use remote storage for persisting shuffle data
  3. SPARK-32504

Shuffle Storage API: Dynamic updates of shuffle metadata

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: Shuffle
    • Labels:
      None

      Description

      When using external storage for shuffles as part of the shuffle storage API mechanism, it is often desirable to update the metadata associated with shuffles that we have enabled plugin systems to implement via https://issues.apache.org/jira/browse/SPARK-31801. For example:

      1. If data is stored in some replicated manner, and the number of replicas is updated - then we want the metadata stored on the driver to reflect the new number of replicas and where they are located.
      2. If data is stored on the mapper's local disk, but is asynchronously backed up to some external storage medium, then we want to know when a backup is available externally.

      To achieve this, we would need to pass a hook to updating the shuffle metadata to the shuffle executor components at the root of the plugin tree on the executor side. The executor would establish an RPC connection with the driver and send messages to update shuffle metadata accordingly.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mcheah Matt Cheah
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: