Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20756

yarn-shuffle jar has references to unshaded guava and contains scala classes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.0.3, 2.1.2, 2.2.0, 2.3.0
    • Component/s: YARN
    • Labels:
      None

      Description

      There are 2 problems with yarn's shuffle jar currently:
      1. It contains shaded guava but it contains references to unshaded classes.

      # Guava is correctly relocated
      >jar -tf common/network-yarn/target/scala-2.11/spark*yarn-shuffle.jar | grep guava | head
      META-INF/maven/com.google.guava/
      META-INF/maven/com.google.guava/guava/
      META-INF/maven/com.google.guava/guava/pom.properties
      META-INF/maven/com.google.guava/guava/pom.xml
      org/spark_project/guava/
      org/spark_project/guava/annotations/
      org/spark_project/guava/annotations/Beta.class
      org/spark_project/guava/annotations/GwtCompatible.class
      org/spark_project/guava/annotations/GwtIncompatible.class
      org/spark_project/guava/annotations/VisibleForTesting.class
      
      # But, there are still references to unshaded guava
      >javap -cp common/network-yarn/target/scala-2.11/spark*yarn-shuffle.jar -c org/apache/spark/network/yarn/YarnShuffleService | grep google
            57: invokestatic  #139                // Method com/google/common/collect/Lists.newArrayList:()Ljava/util/ArrayList;
      

      2. There are references to scala classes in the uber jar:

      jar -tf /opt/src/spark/common/network-yarn/target/scala-2.11/spark-*yarn-shuffle.jar | grep "^scala"
      scala/AnyVal.class
      

      We should fix this.

        Attachments

          Activity

            People

            • Assignee:
              mgrover Mark Grover
              Reporter:
              mgrover Mark Grover
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: