[SPARK-20756] yarn-shuffle jar has references to unshaded guava and contains scala classes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.0.3, 2.1.2, 2.2.0, 2.3.0
• Component/s: Spark Core, YARN
• Assignee: Mark Grover
• Reporter: Mark Grover
• Created: 16/May/17 01:14
• Updated: 17/May/20 18:13
• Resolved: 22/May/17 17:12

Description

There are 2 problems with yarn's shuffle jar currently:
1. It contains shaded guava but it contains references to unshaded classes.

# Guava is correctly relocated
>jar -tf common/network-yarn/target/scala-2.11/spark*yarn-shuffle.jar | grep guava | head
META-INF/maven/com.google.guava/
META-INF/maven/com.google.guava/guava/
META-INF/maven/com.google.guava/guava/pom.properties
META-INF/maven/com.google.guava/guava/pom.xml
org/spark_project/guava/
org/spark_project/guava/annotations/
org/spark_project/guava/annotations/Beta.class
org/spark_project/guava/annotations/GwtCompatible.class
org/spark_project/guava/annotations/GwtIncompatible.class
org/spark_project/guava/annotations/VisibleForTesting.class

# But, there are still references to unshaded guava
>javap -cp common/network-yarn/target/scala-2.11/spark*yarn-shuffle.jar -c org/apache/spark/network/yarn/YarnShuffleService | grep google
      57: invokestatic  #139                // Method com/google/common/collect/Lists.newArrayList:()Ljava/util/ArrayList;

2. There are references to scala classes in the uber jar:

jar -tf /opt/src/spark/common/network-yarn/target/scala-2.11/spark-*yarn-shuffle.jar | grep "^scala"
scala/AnyVal.class

We should fix this.

Issue Links

• links to: [Github] Pull Request #17990 (markgrover)