Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33037

Remove knownManagers hardcoded list

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.7, 3.0.1
    • Fix Version/s: 3.1.0
    • Component/s: Shuffle
    • Labels:
      None

      Description

      Spark has a hardcode list to contain known shuffle managers, which has two values now. It does not contain user's custom shuffle manager which is set through Spark config "spark.shuffle.manager".

       

      We hit issue when set "spark.shuffle.manager" with our own shuffle manager plugin (Uber Remote Shuffle Service implementation, https://github.com/uber/RemoteShuffleService). Other users will hit same issue when they implement their own shuffle manager.

       

      Need to add "spark.shuffle.manager" config value to the known managers list as well.

       

      The know managers list is in code:

      common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java

      private final List<String> knownManagers = Arrays.asList(
        "org.apache.spark.shuffle.sort.SortShuffleManager",
        "org.apache.spark.shuffle.unsafe.UnsafeShuffleManager");

       

       

        Attachments

          Activity

            People

            • Assignee:
              bobyangbo BoYang
              Reporter:
              bobyangbo BoYang
              Shepherd:
              Felix Cheung
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: