[SPARK-20365] Not so accurate classpath format for AM and Containers

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.0, 2.3.0
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Saisai Shao
• Created: 18/Apr/17 07:16
• Updated: 17/May/20 18:14
• Resolved: 01/Jun/17 21:42

Description

In Spark on YARN, when configuring "spark.yarn.jars" with local jars (jars started with "local" scheme), we will get inaccurate classpath for AM and containers. This is because we don't remove "local" scheme when concatenating classpath. It is OK to run because classpath is separated with ":" and java treat "local" as a separate jar. But we could improve it to remove the scheme.

java.class.path = /tmp/hadoop-sshao/nm-local-dir/usercache/sshao/appcache/application_1492057593145_0009/container_1492057593145_0009_01_000003:/tmp/hadoop-sshao/nm-local-dir/usercache/sshao/appcache/application_1492057593145_0009/container_1492057593145_0009_01_000003/__spark_conf__:/tmp/hadoop-sshao/nm-local-dir/usercache/sshao/appcache/application_1492057593145_0009/container_1492057593145_0009_01_000003/__spark_libs__/*:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/activation-1.1.1.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/antlr-2.7.7.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/antlr-runtime-3.4.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/antlr4-runtime-4.5.3.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/aopalliance-1.0.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/aopalliance-repackaged-2.4.0-b34.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/apache-log4j-extras-1.2.17.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/apacheds-i18n-2.0.0-M15.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/apacheds-kerberos-codec-2.0.0-M15.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/api-asn1-api-1.0.0-M20.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/api-util-1.0.0-M20.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/arpack_combined_all-0.1.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/avro-1.7.7.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/avro-ipc-1.7.7-tests.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/avro-ipc-1.7.7.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/avro-mapred-1.7.7-hadoop2.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/base64-2.3.8.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/bcprov-jdk15on-1.51.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/bonecp-0.8.0.RELEASE.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/breeze-macros_2.11-0.12.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/breeze_2.11-0.12.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/calcite-avatica-1.2.0-incubating.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/calcite-core-1.2.0-incubating.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/calcite-linq4j-1.2.0-incubating.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/cglib-2.2.1-v20090111.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/chill-java-0.8.0.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/chill_2.11-0.8.0.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-beanutils-1.7.0.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-beanutils-core-1.8.0.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-cli-1.2.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-codec-1.10.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-collections-3.2.2.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-compiler-3.0.0.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-compress-1.4.1.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-configuration-1.6.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-crypto-1.0.0.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-dbcp-1.4.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-digester-1.8.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-httpclient-3.1.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-io-2.4.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-lang-2.6.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-lang3-3.5.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-logging-1.2.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-math3-3.4.1.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-net-3.1.jar

Issue Links

• links to: [Github] Pull Request #18129 (liyichao)