Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32925

Support push-based shuffle in multiple deployment environments



    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: Shuffle, Spark Core
    • Labels:


      Create this ticket outside of SPARK-30602, since this is outside of the scope of the immediate deliverables in that SPIP. Want to use this ticket to discuss more about how to further improve push-based shuffle in different environments.

      The tasks created under SPARK-30602 would enable push-based shuffle on YARN in a compute/storage colocated cluster. However, there are other deployment environments that are getting more popular these days. We have seen 2 as we discussed with other community members on the idea of push-based shuffle:

      • Spark on K8S in a compute/storage colocated cluster. Because of the limitation of concurrency of read/write of a mounted volume in K8S, multiple executor pods on the same node in a K8S cluster cannot concurrently access the same mounted disk volume. This creates some different requirements for supporting external shuffle service as well as push-based shuffle.
      • Spark on a compute/storage disaggregate cluster. Such a setup is more typical in cloud environments, where the compute cluster has little/no local storage, and the shuffle intermediate data needs to be stored in remote disaggregate storage cluster.

      Want to use this ticket to discuss ways to support push-based shuffle in these different deployment environments.




            • Assignee:
              mshen Min Shen
            • Votes:
              1 Vote for this issue
              7 Start watching this issue


              • Created: