[MAPREDUCE-3389] MRApps loads the 'mrapp-generated-classpath' file with classpath from the build machine

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.23.0, 0.23.1, 2.0.0-alpha
• Fix Version/s: 0.23.1
• Component/s: mrv2
• Labels: bigtop
• Target Version/s: 0.23.1
• Hadoop Flags: Reviewed
• Assignee: Alejandro Abdelnur
• Reporter: Alejandro Abdelnur
• Created: 09/Nov/11 22:30
• Updated: 10/Mar/15 04:32
• Resolved: 05/Dec/11 18:04

Description

The 'mrapp-generated-classpath' file contains the classpath from where Hadoop was build. This classpath is not useful under any circumstances.

For example the content of the 'mrapp-generated-classpath' in my dev environment is:

/Users/tucu/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/Users/tucu/.m2/repository/asm/asm/3.2/asm-3.2.jar:/Users/tucu/.m2/repository/com/cenqua/clover/clover/3.0.2/clover-3.0.2.jar:/Users/tucu/.m2/repository/com/google/guava/guava/r09/guava-r09.jar:/Users/tucu/.m2/repository/com/google/inject/guice/2.0/guice-2.0.jar:/Users/tucu/.m2/repository/com/google/inject/extensions/guice-servlet/2.0/guice-servlet-2.0.jar:/Users/tucu/.m2/repository/com/google/protobuf/protobuf-java/2.4.0a/protobuf-java-2.4.0a.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-core/1.8/jersey-core-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-json/1.8/jersey-json-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-server/1.8/jersey-server-1.8.jar:/Users/tucu/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/Users/tucu/.m2/repository/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar:/Users/tucu/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/tucu/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/tucu/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/tucu/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/Users/tucu/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/Users/tucu/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/tucu/.m2/repository/commons-daemon/commons-daemon/1.0.3/commons-daemon-1.0.3.jar:/Users/tucu/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/tucu/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/Users/tucu/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/Users/tucu/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:/Users/tucu/.m2/repository/commons-lang/commons-lang/2.5/commons-lang-2.5.jar:/Users/tucu/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/tucu/.m2/repository/commons-logging/commons-logging-api/1.1/commons-logging-api-1.1.jar:/Users/tucu/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar:/Users/tucu/.m2/repository/hsqldb/hsqldb/1.8.0.7/hsqldb-1.8.0.7.jar:/Users/tucu/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/tucu/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:/Users/tucu/.m2/repository/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar:/Users/tucu/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/Users/tucu/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/tucu/.m2/repository/jdiff/jdiff/1.0.9/jdiff-1.0.9.jar:/Users/tucu/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/Users/tucu/.m2/repository/junit/junit/4.8.2/junit-4.8.2.jar:/Users/tucu/.m2/repository/log4j/log4j/1.2.15/log4j-1.2.15.jar:/Users/tucu/.m2/repository/net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar:/Users/tucu/.m2/repository/net/sf/kosmosfs/kfs/0.3/kfs-0.3.jar:/Users/tucu/.m2/repository/org/apache/avro/avro/1.5.3/avro-1.5.3.jar:/Users/tucu/.m2/repository/org/apache/avro/avro-ipc/1.5.3/avro-ipc-1.5.3.jar:/Users/tucu/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:/Users/tucu/src/apache/hadoop/git/hadoop-common-project/hadoop-annotations/target/hadoop-annotations-0.24.0-SNAPSHOT.jar:/Users/tucu/src/apache/hadoop/git/hadoop-common-project/hadoop-auth/target/hadoop-auth-0.24.0-SNAPSHOT.jar:/Users/tucu/src/apache/hadoop/git/hadoop-common-project/hadoop-common/target/hadoop-common-0.24.0-SNAPSHOT.jar:/Users/tucu/src/apache/hadoop/git/hadoop-common-project/hadoop-common/target/hadoop-common-0.24.0-SNAPSHOT-tests.jar:/Users/tucu/src/apache/hadoop/git/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-0.24.0-SNAPSHOT.jar:/Users/tucu/src/apache/hadoop/git/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/target/hadoop-mapreduce-client-common-0.24.0-SNAPSHOT.jar:/Users/tucu/src/apache/hadoop/git/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/hadoop-mapreduce-client-core-0.24.0-SNAPSHOT.jar:/Users/tucu/src/apache/hadoop/git/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/target/hadoop-mapreduce-client-shuffle-0.24.0-SNAPSHOT.jar:/Users/tucu/src/apache/hadoop/git/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/target/hadoop-yarn-api-0.24.0-SNAPSHOT.jar:/Users/tucu/src/apache/hadoop/git/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/target/hadoop-yarn-common-0.24.0-SNAPSHOT.jar:/Users/tucu/src/apache/hadoop/git/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/target/hadoop-yarn-common-0.24.0-SNAPSHOT-tests.jar:/Users/tucu/src/apache/hadoop/git/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/target/hadoop-yarn-server-common-0.24.0-SNAPSHOT.jar:/Users/tucu/src/apache/hadoop/git/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/hadoop-yarn-server-nodemanager-0.24.0-SNAPSHOT.jar:/Users/tucu/src/apache/hadoop/git/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/hadoop-yarn-server-resourcemanager-0.24.0-SNAPSHOT.jar:/Users/tucu/src/apache/hadoop/git/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/hadoop-yarn-server-resourcemanager-0.24.0-SNAPSHOT-tests.jar:/Users/tucu/src/apache/hadoop/git/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/target/hadoop-yarn-server-web-proxy-0.24.0-SNAPSHOT.jar:/Users/tucu/.m2/repository/org/apache/velocity/velocity/1.7/velocity-1.7.jar:/Users/tucu/.m2/repository/org/apache/zookeeper/zookeeper/3.3.1/zookeeper-3.3.1.jar:/Users/tucu/.m2/repository/org/aspectj/aspectjrt/1.6.5/aspectjrt-1.6.5.jar:/Users/tucu/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.7.1/jackson-core-asl-1.7.1.jar:/Users/tucu/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.7.1/jackson-jaxrs-1.7.1.jar:/Users/tucu/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.1/jackson-mapper-asl-1.7.1.jar:/Users/tucu/.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar:/Users/tucu/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/Users/tucu/.m2/repository/org/eclipse/jdt/core/3.1.1/core-3.1.1.jar:/Users/tucu/.m2/repository/org/jboss/netty/netty/3.2.3.Final/netty-3.2.3.Final.jar:/Users/tucu/.m2/repository/org/mockito/mockito-all/1.8.5/mockito-all-1.8.5.jar:/Users/tucu/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar:/Users/tucu/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/Users/tucu/.m2/repository/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar:/Users/tucu/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar:/Users/tucu/.m2/repository/org/xerial/snappy/snappy-java/1.0.3.2/snappy-java-1.0.3.2.jar:/Users/tucu/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/tucu/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/Users/tucu/.m2/repository/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar:/Users/tucu/.m2/repository/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar:/Users/tucu/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar

Issue Links

• duplicates: MAPREDUCE-3428 MR AppMaster CLASSPATH is dependent on the compile-time environment
• is depended upon by: PIG-2347 Fix Pig Unit tests for hadoop 23
• is related to: MAPREDUCE-3505 yarn APPLICATION_CLASSPATH needs to be overridable
• is related to: MAPREDUCE-3509 improve MRApp generated classpath
• relates to: MAPREDUCE-3616 Thread pool for launching containers in MR AM not expanding as expected
• relates to: MAPREDUCE-3087 CLASSPATH not the same after MAPREDUCE-2880