Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6577

MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
    • Component/s: mr-am
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:

      2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      

      As a result, any code that needs the hadoop native library in the MR AM will fail. For example, an uber-AM with lz4 compression for the mapper task will fail:

      2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local (uberized) 'child' : java.lang.RuntimeException: native lz4 library not available
      	at org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
      	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
      	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
      	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)
      	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)
      	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
      	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
      	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
      	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
      	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
      	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      
      1. MAPREDUCE-6577.01.patch
        9 kB
        Sangjin Lee
      2. MAPREDUCE-6577.02.patch
        9 kB
        Sangjin Lee
      3. MAPREDUCE-6577.03.patch
        13 kB
        Sangjin Lee
      4. MAPREDUCE-6577.04.patch
        13 kB
        Sangjin Lee

        Issue Links

          Activity

          Hide
          sjlee0 Sangjin Lee added a comment -

          I believe this is caused by MAPREDUCE-6021. Previously, at least in the case of the default container executor the MR AM basically inherited the environment variables from the parent process (node manager), so it got LD_LIBRARY_PATH that contained ${HADOOP_HOME}/lib/native.

          In 2.6 and later, LD_LIBRARY_PATH is set to $PWD, and ${HADOOP_HOME}/lib/native has effectively been removed from LD_LIBRARY_PATH.

          The intent of MAPREDUCE-6021 was to handle the MR AM in the same way as non-AM tasks. However, non-AM tasks have the default admin.user.env value that provides ${HADOOP_HOME}/lib/native (in TaskAttemptImpl.java). The AM does not have the corresponding default, thus the symmetry is broken there.

          Show
          sjlee0 Sangjin Lee added a comment - I believe this is caused by MAPREDUCE-6021 . Previously, at least in the case of the default container executor the MR AM basically inherited the environment variables from the parent process (node manager), so it got LD_LIBRARY_PATH that contained ${HADOOP_HOME}/lib/native. In 2.6 and later, LD_LIBRARY_PATH is set to $PWD, and ${HADOOP_HOME}/lib/native has effectively been removed from LD_LIBRARY_PATH. The intent of MAPREDUCE-6021 was to handle the MR AM in the same way as non-AM tasks. However, non-AM tasks have the default admin.user.env value that provides ${HADOOP_HOME}/lib/native (in TaskAttemptImpl.java). The AM does not have the corresponding default, thus the symmetry is broken there.
          Hide
          sjlee0 Sangjin Lee added a comment -

          MAPREDUCE-5799 is very close to this JIRA, but since 2.6.0 the issue has gotten worse and more generic (MR AM cannot load the main hadoop native library). Closed that one in favor of this more current description of the problem.

          Show
          sjlee0 Sangjin Lee added a comment - MAPREDUCE-5799 is very close to this JIRA, but since 2.6.0 the issue has gotten worse and more generic (MR AM cannot load the main hadoop native library). Closed that one in favor of this more current description of the problem.
          Hide
          sjlee0 Sangjin Lee added a comment -

          My proposal would be to handle this in the same way as the non-AM tasks; i.e. provide the same default value for yarn.app.mapreduce.am.admin.user.env. That would also complete the intent of MAPREDUCE-6021.

          Show
          sjlee0 Sangjin Lee added a comment - My proposal would be to handle this in the same way as the non-AM tasks; i.e. provide the same default value for yarn.app.mapreduce.am.admin.user.env. That would also complete the intent of MAPREDUCE-6021 .
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for the report and analysis, Sangjin! I agree there should be a default to match the behavior with tasks – sorry this was missed in MAPREDUCE-6021. We didn't catch it on our clusters because we set the admin properties in order to use a MapReduce framework via HDFS. We explicitly do not want to rely on any Hadoop bits from the nodes to make rolling upgrades more robust. Otherwise we could mix versions of Java and JNI bits and lead to weird behaviors.

          Show
          jlowe Jason Lowe added a comment - Thanks for the report and analysis, Sangjin! I agree there should be a default to match the behavior with tasks – sorry this was missed in MAPREDUCE-6021 . We didn't catch it on our clusters because we set the admin properties in order to use a MapReduce framework via HDFS. We explicitly do not want to rely on any Hadoop bits from the nodes to make rolling upgrades more robust. Otherwise we could mix versions of Java and JNI bits and lead to weird behaviors.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.1.

          Added the default for the AM admin user env (which has the same value as the non-AM admin user env).

          Also did a small amount of yak-shaving. I suspect that the lib path handling on Windows is not quite right (PWD was not being added for Windows). I tried to make it a little more cross-platform there.

          Folks who use Windows (Chris Nauroth?), I'd greatly appreciate it if you could review the patch from that perspective.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.1. Added the default for the AM admin user env (which has the same value as the non-AM admin user env). Also did a small amount of yak-shaving. I suspect that the lib path handling on Windows is not quite right (PWD was not being added for Windows). I tried to make it a little more cross-platform there. Folks who use Windows ( Chris Nauroth ?), I'd greatly appreciate it if you could review the patch from that perspective.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 9s trunk passed
          +1 compile 3m 20s trunk passed with JDK v1.8.0_66
          +1 compile 3m 19s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 1m 34s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          -1 findbugs 1m 16s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
          +1 javadoc 0m 54s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 59s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 21s the patch passed
          +1 compile 3m 6s the patch passed with JDK v1.8.0_66
          -1 javac 6m 51s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 360, now 360).
          +1 javac 3m 6s the patch passed
          +1 compile 3m 18s the patch passed with JDK v1.7.0_91
          -1 javac 10m 9s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91 with JDK v1.7.0_91 generated 2 new issues (was 365, now 365).
          +1 javac 3m 18s the patch passed
          -1 checkstyle 0m 21s Patch generated 5 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client (total was 794, now 796).
          +1 mvnsite 1m 29s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 52s the patch passed
          +1 javadoc 0m 52s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 58s the patch passed with JDK v1.7.0_91
          +1 unit 2m 10s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
          -1 unit 9m 42s hadoop-mapreduce-client-app in the patch failed with JDK v1.8.0_66.
          -1 unit 108m 6s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66.
          +1 unit 2m 34s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
          +1 unit 10m 21s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_91.
          -1 unit 108m 28s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 31s Patch generated 16 ASF License warnings.
          280m 4s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.app.job.impl.TestJobImpl
            hadoop.mapred.TestNetworkedJob
            hadoop.mapreduce.v2.TestMRJobsWithProfiler
          JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778317/MAPREDUCE-6577.01.patch
          JIRA Issue MAPREDUCE-6577
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6c1697152acd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4e7d32c
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
          javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66.txt
          javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 9s trunk passed +1 compile 3m 20s trunk passed with JDK v1.8.0_66 +1 compile 3m 19s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 20s trunk passed +1 mvnsite 1m 34s trunk passed +1 mvneclipse 0m 43s trunk passed -1 findbugs 1m 16s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 54s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 59s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 21s the patch passed +1 compile 3m 6s the patch passed with JDK v1.8.0_66 -1 javac 6m 51s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 360, now 360). +1 javac 3m 6s the patch passed +1 compile 3m 18s the patch passed with JDK v1.7.0_91 -1 javac 10m 9s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91 with JDK v1.7.0_91 generated 2 new issues (was 365, now 365). +1 javac 3m 18s the patch passed -1 checkstyle 0m 21s Patch generated 5 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client (total was 794, now 796). +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 52s the patch passed +1 javadoc 0m 52s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 58s the patch passed with JDK v1.7.0_91 +1 unit 2m 10s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. -1 unit 9m 42s hadoop-mapreduce-client-app in the patch failed with JDK v1.8.0_66. -1 unit 108m 6s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66. +1 unit 2m 34s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. +1 unit 10m 21s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_91. -1 unit 108m 28s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 31s Patch generated 16 ASF License warnings. 280m 4s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.app.job.impl.TestJobImpl   hadoop.mapred.TestNetworkedJob   hadoop.mapreduce.v2.TestMRJobsWithProfiler JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778317/MAPREDUCE-6577.01.patch JIRA Issue MAPREDUCE-6577 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6c1697152acd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4e7d32c findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66.txt javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6213/console This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          All the issues that are flagged by jenkins (findbugs, javac, checkstyle, and unit tests) appear to be existing issues that are unrelated to the patch.

          I'd appreciate your review on the patch. Thanks!

          Show
          sjlee0 Sangjin Lee added a comment - All the issues that are flagged by jenkins (findbugs, javac, checkstyle, and unit tests) appear to be existing issues that are unrelated to the patch. I'd appreciate your review on the patch. Thanks!
          Hide
          jlowe Jason Lowe added a comment - - edited

          The environment variable reference should be like this:

            public static final String DEFAULT_MR_AM_ADMIN_USER_ENV =
                Shell.WINDOWS ?
                    "PATH={{PATH}};{{HADOOP_COMMON_HOME}}\\" + RELATIVE_LIB_PATH :
                    "LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/" + RELATIVE_LIB_PATH;
          

          Otherwise it will be expanded by the job client's version of those environment variables rather than the nodemanager. The client could have a totally bogus PATH or HADOOP_COMMON_HOME setting relative to what the cluster nodes need. It works for tasks today because it's being setup by the AM, and the AM is inheriting the nodemanager's variables instead of the job client's.

          Otherwise patch looks good. I'm not a Windows expert, but those changes look OK to me.

          Show
          jlowe Jason Lowe added a comment - - edited The environment variable reference should be like this: public static final String DEFAULT_MR_AM_ADMIN_USER_ENV = Shell.WINDOWS ? "PATH={{PATH}};{{HADOOP_COMMON_HOME}}\\" + RELATIVE_LIB_PATH : "LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/" + RELATIVE_LIB_PATH; Otherwise it will be expanded by the job client's version of those environment variables rather than the nodemanager. The client could have a totally bogus PATH or HADOOP_COMMON_HOME setting relative to what the cluster nodes need. It works for tasks today because it's being setup by the AM, and the AM is inheriting the nodemanager's variables instead of the job client's. Otherwise patch looks good. I'm not a Windows expert, but those changes look OK to me.
          Hide
          jlowe Jason Lowe added a comment -

          Actually instead of hardcoding the braces it would be cleaner to use the crossPlatformify method on those variables which accomplishes the same thing.

          Show
          jlowe Jason Lowe added a comment - Actually instead of hardcoding the braces it would be cleaner to use the crossPlatformify method on those variables which accomplishes the same thing.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for the pointer Jason Lowe! MRApps.crossPlatformifyMREnv() does that only if mapreduce.app-submission.cross-platform is true (which is false by default)? So it looks like it'd be safer if I used {{ and }} directly, no? Also, the method requires a Configuration object and an Environment object, so I'm not sure if that works in MRJobConfig.

          Show
          sjlee0 Sangjin Lee added a comment - Thanks for the pointer Jason Lowe ! MRApps.crossPlatformifyMREnv() does that only if mapreduce.app-submission.cross-platform is true (which is false by default)? So it looks like it'd be safer if I used {{ and }} directly, no? Also, the method requires a Configuration object and an Environment object, so I'm not sure if that works in MRJobConfig.
          Hide
          jlowe Jason Lowe added a comment -

          Sorry, I should have been more clear since the method names are so similar. I was referring to Apps.crossPlatformify instead of MRApps.crossPlatformifyMREnv. The former is already being used in MRJobConfig by DEFAULT_MAPREDUCE_CROSS_PLATFORM_APPLICATION_CLASSPATH.

          Show
          jlowe Jason Lowe added a comment - Sorry, I should have been more clear since the method names are so similar. I was referring to Apps.crossPlatformify instead of MRApps.crossPlatformifyMREnv. The former is already being used in MRJobConfig by DEFAULT_MAPREDUCE_CROSS_PLATFORM_APPLICATION_CLASSPATH.
          Hide
          jlowe Jason Lowe added a comment -

          And since those won't be expanded, we can probably eliminate the RELATIVE_LIB_PATH exposure and just have the unit test check if DEFAULT_MR_AM_ADMIN_USER_ENV appears in the launch context since the values shouldn't be expanded on the client.

          Show
          jlowe Jason Lowe added a comment - And since those won't be expanded, we can probably eliminate the RELATIVE_LIB_PATH exposure and just have the unit test check if DEFAULT_MR_AM_ADMIN_USER_ENV appears in the launch context since the values shouldn't be expanded on the client.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for the clarification. Yes, that's what I'm going to do now that the variables are not going to be expanded.

          Show
          sjlee0 Sangjin Lee added a comment - Thanks for the clarification. Yes, that's what I'm going to do now that the variables are not going to be expanded.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.2 that addresses Jason's review comments.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.2 that addresses Jason's review comments.
          Hide
          jlowe Jason Lowe added a comment -

          +1 lgtm, pending Jenkins

          Show
          jlowe Jason Lowe added a comment - +1 lgtm, pending Jenkins
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 21s trunk passed
          +1 compile 3m 17s trunk passed with JDK v1.8.0_66
          +1 compile 3m 25s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 1m 35s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          -1 findbugs 1m 19s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
          +1 javadoc 0m 55s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 1s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 22s the patch passed
          +1 compile 3m 13s the patch passed with JDK v1.8.0_66
          -1 javac 7m 2s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 360, now 360).
          +1 javac 3m 13s the patch passed
          +1 compile 3m 23s the patch passed with JDK v1.7.0_91
          -1 javac 10m 25s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91 with JDK v1.7.0_91 generated 2 new issues (was 365, now 365).
          +1 javac 3m 23s the patch passed
          -1 checkstyle 0m 21s Patch generated 4 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client (total was 794, now 795).
          +1 mvnsite 1m 35s the patch passed
          +1 mvneclipse 0m 43s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 59s the patch passed
          +1 javadoc 0m 54s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 0s the patch passed with JDK v1.7.0_91
          +1 unit 2m 9s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
          -1 unit 9m 52s hadoop-mapreduce-client-app in the patch failed with JDK v1.8.0_66.
          -1 unit 123m 10s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66.
          +1 unit 2m 34s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
          +1 unit 10m 35s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_91.
          -1 unit 107m 38s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 32s Patch generated 18 ASF License warnings.
          295m 41s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.app.job.impl.TestJobImpl
            hadoop.mapreduce.v2.TestMRJobsWithProfiler
            hadoop.mapred.TestNetworkedJob
            hadoop.mapred.TestMiniMRChildTask
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl
          JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob
            hadoop.mapred.TestMiniMRChildTask



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778529/MAPREDUCE-6577.02.patch
          JIRA Issue MAPREDUCE-6577
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5449eb9a6656 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e63388f
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
          javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66.txt
          javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 21s trunk passed +1 compile 3m 17s trunk passed with JDK v1.8.0_66 +1 compile 3m 25s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 35s trunk passed +1 mvneclipse 0m 43s trunk passed -1 findbugs 1m 19s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 55s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 1s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 22s the patch passed +1 compile 3m 13s the patch passed with JDK v1.8.0_66 -1 javac 7m 2s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 360, now 360). +1 javac 3m 13s the patch passed +1 compile 3m 23s the patch passed with JDK v1.7.0_91 -1 javac 10m 25s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91 with JDK v1.7.0_91 generated 2 new issues (was 365, now 365). +1 javac 3m 23s the patch passed -1 checkstyle 0m 21s Patch generated 4 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client (total was 794, now 795). +1 mvnsite 1m 35s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 59s the patch passed +1 javadoc 0m 54s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 0s the patch passed with JDK v1.7.0_91 +1 unit 2m 9s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. -1 unit 9m 52s hadoop-mapreduce-client-app in the patch failed with JDK v1.8.0_66. -1 unit 123m 10s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66. +1 unit 2m 34s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. +1 unit 10m 35s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_91. -1 unit 107m 38s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 32s Patch generated 18 ASF License warnings. 295m 41s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.app.job.impl.TestJobImpl   hadoop.mapreduce.v2.TestMRJobsWithProfiler   hadoop.mapred.TestNetworkedJob   hadoop.mapred.TestMiniMRChildTask JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob   hadoop.mapred.TestMiniMRChildTask Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778529/MAPREDUCE-6577.02.patch JIRA Issue MAPREDUCE-6577 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5449eb9a6656 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e63388f findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66.txt javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6217/console This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          It looks like TestMiniMRChildTask#testMapRedExecutionEnv() needs updating. Please hold off on committing it until I update the patch. Thanks.

          Show
          sjlee0 Sangjin Lee added a comment - It looks like TestMiniMRChildTask#testMapRedExecutionEnv() needs updating. Please hold off on committing it until I update the patch. Thanks.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.3.

          Fixed TestMiniMRChildTask#testMapRedExecutionEnv(). Removed the first part of the test as it is not meaningful to test the HADOOP_COMMON_HOME substitution as it is no longer substituted with the late binding.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.3. Fixed TestMiniMRChildTask#testMapRedExecutionEnv(). Removed the first part of the test as it is not meaningful to test the HADOOP_COMMON_HOME substitution as it is no longer substituted with the late binding.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 10s trunk passed
          +1 compile 3m 17s trunk passed with JDK v1.8.0_66
          +1 compile 3m 21s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 1m 38s trunk passed
          +1 mvneclipse 0m 45s trunk passed
          -1 findbugs 1m 18s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
          +1 javadoc 0m 55s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 5s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 25s the patch passed
          +1 compile 3m 16s the patch passed with JDK v1.8.0_66
          -1 javac 7m 11s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66 with JDK v1.8.0_66 generated 10 new issues (was 360, now 360).
          +1 javac 3m 16s the patch passed
          +1 compile 3m 19s the patch passed with JDK v1.7.0_91
          -1 javac 10m 30s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91 with JDK v1.7.0_91 generated 11 new issues (was 365, now 365).
          +1 javac 3m 19s the patch passed
          -1 checkstyle 0m 20s Patch generated 4 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client (total was 794, now 795).
          +1 mvnsite 1m 32s the patch passed
          +1 mvneclipse 0m 42s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 52s the patch passed
          +1 javadoc 0m 52s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 1s the patch passed with JDK v1.7.0_91
          -1 unit 2m 4s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_66.
          -1 unit 9m 37s hadoop-mapreduce-client-app in the patch failed with JDK v1.8.0_66.
          -1 unit 106m 56s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66.
          +1 unit 2m 30s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
          +1 unit 10m 13s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_91.
          -1 unit 108m 50s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 31s Patch generated 17 ASF License warnings.
          279m 36s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.tools.TestCLI
            hadoop.mapreduce.v2.app.job.impl.TestJobImpl
            hadoop.mapreduce.v2.TestMRJobsWithProfiler
            hadoop.mapred.TestNetworkedJob
          JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778679/MAPREDUCE-6577.03.patch
          JIRA Issue MAPREDUCE-6577
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0d7c46cb6ae2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8652cce
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
          javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66.txt
          javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 10s trunk passed +1 compile 3m 17s trunk passed with JDK v1.8.0_66 +1 compile 3m 21s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 38s trunk passed +1 mvneclipse 0m 45s trunk passed -1 findbugs 1m 18s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 55s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 5s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 25s the patch passed +1 compile 3m 16s the patch passed with JDK v1.8.0_66 -1 javac 7m 11s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66 with JDK v1.8.0_66 generated 10 new issues (was 360, now 360). +1 javac 3m 16s the patch passed +1 compile 3m 19s the patch passed with JDK v1.7.0_91 -1 javac 10m 30s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91 with JDK v1.7.0_91 generated 11 new issues (was 365, now 365). +1 javac 3m 19s the patch passed -1 checkstyle 0m 20s Patch generated 4 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client (total was 794, now 795). +1 mvnsite 1m 32s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 52s the patch passed +1 javadoc 0m 52s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 1s the patch passed with JDK v1.7.0_91 -1 unit 2m 4s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_66. -1 unit 9m 37s hadoop-mapreduce-client-app in the patch failed with JDK v1.8.0_66. -1 unit 106m 56s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66. +1 unit 2m 30s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. +1 unit 10m 13s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_91. -1 unit 108m 50s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 31s Patch generated 17 ASF License warnings. 279m 36s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.tools.TestCLI   hadoop.mapreduce.v2.app.job.impl.TestJobImpl   hadoop.mapreduce.v2.TestMRJobsWithProfiler   hadoop.mapred.TestNetworkedJob JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778679/MAPREDUCE-6577.03.patch JIRA Issue MAPREDUCE-6577 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0d7c46cb6ae2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8652cce findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66.txt javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6219/console This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          The latest patch (v.3) should now be good to review.

          I was thrown off for a bit because Mac OS (on which I was testing some of these unit tests) doesn't seem to handle LD_LIBRARY_PATH in the same way as Linux. In fact, the trunk version of TestMiniMRChildTask does not pass on Mac. But on Linux, the test is fine before and after the patch.

          I would appreciate your review. Thanks!

          Show
          sjlee0 Sangjin Lee added a comment - The latest patch (v.3) should now be good to review. I was thrown off for a bit because Mac OS (on which I was testing some of these unit tests) doesn't seem to handle LD_LIBRARY_PATH in the same way as Linux. In fact, the trunk version of TestMiniMRChildTask does not pass on Mac. But on Linux, the test is fine before and after the patch. I would appreciate your review. Thanks!
          Hide
          laurentgo Laurent Goujon added a comment -

          AFAIK (and confirmed by dyld(1) man page), MacOS uses DYLD_LIBRARY_PATH

          Show
          laurentgo Laurent Goujon added a comment - AFAIK (and confirmed by dyld(1) man page), MacOS uses DYLD_LIBRARY_PATH
          Hide
          cnauroth Chris Nauroth added a comment -

          FYI, HDFS-3296 is an incomplete, uncommitted patch that adds DYLD_LIBRARY_PATH to the environment to enable more of the native tests on Mac.

          Show
          cnauroth Chris Nauroth added a comment - FYI, HDFS-3296 is an incomplete, uncommitted patch that adds DYLD_LIBRARY_PATH to the environment to enable more of the native tests on Mac.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks Laurent Goujon and Chris Nauroth for the pointers! Yes, I am aware that Mac uses DYLD_LIBRARY_PATH, but there seem to be more quirks (perhaps version-dependent?) that mess with that. Since it's an existing larger issue, I think it's safe to say it's outside the scope of this JIRA.

          Show
          sjlee0 Sangjin Lee added a comment - Thanks Laurent Goujon and Chris Nauroth for the pointers! Yes, I am aware that Mac uses DYLD_LIBRARY_PATH, but there seem to be more quirks (perhaps version-dependent?) that mess with that. Since it's an existing larger issue, I think it's safe to say it's outside the scope of this JIRA.
          Hide
          jlowe Jason Lowe added a comment -

          +1 lgtm. I'll commit this tomorrow if there are no further comments.

          Show
          jlowe Jason Lowe added a comment - +1 lgtm. I'll commit this tomorrow if there are no further comments.
          Hide
          cnauroth Chris Nauroth added a comment -

          The current version of the patch has introduced test failures on Windows in TestMiniMRChildTask and TestMiniMRClientCluster. Can you please hold off committing? I'll try to take a closer look at root cause tomorrow.

          Show
          cnauroth Chris Nauroth added a comment - The current version of the patch has introduced test failures on Windows in TestMiniMRChildTask and TestMiniMRClientCluster . Can you please hold off committing? I'll try to take a closer look at root cause tomorrow.
          Hide
          sjlee0 Sangjin Lee added a comment -

          No worries. Thanks for checking. It's rather difficult to acquire a Windows machine that's ready to build hadoop.

          Show
          sjlee0 Sangjin Lee added a comment - No worries. Thanks for checking. It's rather difficult to acquire a Windows machine that's ready to build hadoop.
          Hide
          cnauroth Chris Nauroth added a comment -

          It's rather difficult to acquire a Windows machine that's ready to build hadoop.

          Any committer on any Apache project has access to a free MSDN license. This is sufficient to run a Windows VM that can build Hadoop and run the JUnit tests. Here is an email thread where I discussed this in more detail:

          http://s.apache.org/YhB

          Show
          cnauroth Chris Nauroth added a comment - It's rather difficult to acquire a Windows machine that's ready to build hadoop. Any committer on any Apache project has access to a free MSDN license. This is sufficient to run a Windows VM that can build Hadoop and run the JUnit tests. Here is an email thread where I discussed this in more detail: http://s.apache.org/YhB
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks Chris. I actually stated down my path on setting that up, but I was having tons of trouble setting up the Visual Studio (or Windows SDK?). Even with the native build turned off, I kept getting errors around msbuild and Visual Studio. I tried all kinds of different versions (10, 2013, 2015) with no success. I'll give it one more shot.

          Show
          sjlee0 Sangjin Lee added a comment - Thanks Chris. I actually stated down my path on setting that up, but I was having tons of trouble setting up the Visual Studio (or Windows SDK?). Even with the native build turned off, I kept getting errors around msbuild and Visual Studio. I tried all kinds of different versions (10, 2013, 2015) with no success. I'll give it one more shot.
          Hide
          sjlee0 Sangjin Lee added a comment -

          After looking into the Windows side of things for a while, I realize that there are bigger issues to address on that side that go beyond the scope of this JIRA. Also, Windows isn't exactly broken in the same manner as Linux is by MAPREDUCE-6021 as it didn't really touch Windows.

          I filed a separate JIRA to address the Windows parity and less major issues (MAPREDUCE-6588). I propose we tackle the Windows aspect in that JIRA.

          I'll update the patch shortly to preserve the Windows behavior as it is today.

          Show
          sjlee0 Sangjin Lee added a comment - After looking into the Windows side of things for a while, I realize that there are bigger issues to address on that side that go beyond the scope of this JIRA. Also, Windows isn't exactly broken in the same manner as Linux is by MAPREDUCE-6021 as it didn't really touch Windows. I filed a separate JIRA to address the Windows parity and less major issues ( MAPREDUCE-6588 ). I propose we tackle the Windows aspect in that JIRA. I'll update the patch shortly to preserve the Windows behavior as it is today.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.4.

          This keeps the Windows behavior identical to the current trunk. As a result, some of the code changes are slightly uglier.

          Another interesting change: as part of testing this pretty extensively, I stumbled onto the fact that HADOOP_COMMON_HOME was not being set in the hadoop unit tests. There are a lot of places where HADOOP_COMMON_HOME is used (including this one), and I suppose this problem was masked because most paths were being inherited from the node manager (mini-YARN cluster).

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.4. This keeps the Windows behavior identical to the current trunk. As a result, some of the code changes are slightly uglier. Another interesting change: as part of testing this pretty extensively, I stumbled onto the fact that HADOOP_COMMON_HOME was not being set in the hadoop unit tests. There are a lot of places where HADOOP_COMMON_HOME is used (including this one), and I suppose this problem was masked because most paths were being inherited from the node manager (mini-YARN cluster).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 56s trunk passed
          +1 compile 9m 27s trunk passed with JDK v1.8.0_66
          +1 compile 10m 5s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 5s trunk passed
          +1 mvnsite 1m 23s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          -1 findbugs 0m 11s branch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml)
          -1 findbugs 1m 21s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
          +1 javadoc 0m 49s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 54s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 10s the patch passed
          +1 compile 9m 22s the patch passed with JDK v1.8.0_66
          +1 javac 9m 22s the patch passed
          +1 compile 9m 58s the patch passed with JDK v1.7.0_91
          +1 javac 9m 58s the patch passed
          -1 checkstyle 1m 5s Patch generated 2 new checkstyle issues in root (total was 548, now 549).
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          -1 findbugs 0m 11s patch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml)
          +1 javadoc 0m 49s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 59s the patch passed with JDK v1.7.0_91
          +1 unit 0m 10s hadoop-project in the patch passed with JDK v1.8.0_66.
          -1 unit 2m 8s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_66.
          -1 unit 102m 18s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 16s hadoop-project in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 32s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
          -1 unit 107m 40s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 33s Patch generated 17 ASF License warnings.
          279m 2s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.tools.TestCLI
            hadoop.mapreduce.v2.TestMRJobsWithProfiler
            hadoop.mapred.TestNetworkedJob
          JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779362/MAPREDUCE-6577.04.patch
          JIRA Issue MAPREDUCE-6577
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux e15782e00999 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0af492b
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-project U: .
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 56s trunk passed +1 compile 9m 27s trunk passed with JDK v1.8.0_66 +1 compile 10m 5s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 5s trunk passed +1 mvnsite 1m 23s trunk passed +1 mvneclipse 0m 43s trunk passed -1 findbugs 0m 11s branch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml) -1 findbugs 1m 21s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 49s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 54s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 10s the patch passed +1 compile 9m 22s the patch passed with JDK v1.8.0_66 +1 javac 9m 22s the patch passed +1 compile 9m 58s the patch passed with JDK v1.7.0_91 +1 javac 9m 58s the patch passed -1 checkstyle 1m 5s Patch generated 2 new checkstyle issues in root (total was 548, now 549). +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 0m 11s patch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml) +1 javadoc 0m 49s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 59s the patch passed with JDK v1.7.0_91 +1 unit 0m 10s hadoop-project in the patch passed with JDK v1.8.0_66. -1 unit 2m 8s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_66. -1 unit 102m 18s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66. +1 unit 0m 16s hadoop-project in the patch passed with JDK v1.7.0_91. +1 unit 2m 32s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. -1 unit 107m 40s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 33s Patch generated 17 ASF License warnings. 279m 2s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.tools.TestCLI   hadoop.mapreduce.v2.TestMRJobsWithProfiler   hadoop.mapred.TestNetworkedJob JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779362/MAPREDUCE-6577.04.patch JIRA Issue MAPREDUCE-6577 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux e15782e00999 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0af492b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-project U: . Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6231/console This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          We really need to make a bigger effort to remove all of these DEFAULT_blah env vars so that we stop spewing all over the shell's environment name space.

          Show
          aw Allen Wittenauer added a comment - We really need to make a bigger effort to remove all of these DEFAULT_blah env vars so that we stop spewing all over the shell's environment name space.
          Hide
          cnauroth Chris Nauroth added a comment -

          Sangjin Lee, thank you for following up. I tested patch v04, and I no longer see those failures on Windows. The failure in TestNetworkedJob occurs even without this patch, and that's tracked in MAPREDUCE-6579.

          +1 for patch v04.

          Show
          cnauroth Chris Nauroth added a comment - Sangjin Lee , thank you for following up. I tested patch v04, and I no longer see those failures on Windows. The failure in TestNetworkedJob occurs even without this patch, and that's tracked in MAPREDUCE-6579 . +1 for patch v04.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks Chris Nauroth! Jason Lowe, do let me know if you're +1 on the latest patch. I can commit it then.

          Show
          sjlee0 Sangjin Lee added a comment - Thanks Chris Nauroth ! Jason Lowe , do let me know if you're +1 on the latest patch. I can commit it then.
          Hide
          jlowe Jason Lowe added a comment -

          I'm +1 for patch v04 as well. Feel free to commit.

          Show
          jlowe Jason Lowe added a comment - I'm +1 for patch v04 as well. Feel free to commit.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9052 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9052/)
          MAPREDUCE-6577. MR AM unable to load native library without (sjlee: rev f6f16118d38fcfe3b724f05fad752cb223f441ec)

          • hadoop-project/pom.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9052 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9052/ ) MAPREDUCE-6577 . MR AM unable to load native library without (sjlee: rev f6f16118d38fcfe3b724f05fad752cb223f441ec) hadoop-project/pom.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9053 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9053/)
          MAPREDUCE-6577 addendum. Moved the changelog to 2.6.4 which is the (sjlee: rev 355c0ce7237c5208557bb32ec0423819bb9b47b3)

          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9053 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9053/ ) MAPREDUCE-6577 addendum. Moved the changelog to 2.6.4 which is the (sjlee: rev 355c0ce7237c5208557bb32ec0423819bb9b47b3) hadoop-mapreduce-project/CHANGES.txt
          Hide
          sjlee0 Sangjin Lee added a comment -

          Committed the patch to trunk, branch-2, branch-2.7, and branch-2.6. Thanks Jason Lowe and Chris Nauroth for your reviews and suggestions!

          Show
          sjlee0 Sangjin Lee added a comment - Committed the patch to trunk, branch-2, branch-2.7, and branch-2.6. Thanks Jason Lowe and Chris Nauroth for your reviews and suggestions!
          Hide
          jlowe Jason Lowe added a comment -

          Thanks, Sangjin! Could you also commit this to branch-2.8? I find it odd that the fix is important enough to backport to 2.6 and 2.7 but not important enough for 2.8. Users updating from those older releases to 2.8 will accidentally drop this fix.

          Show
          jlowe Jason Lowe added a comment - Thanks, Sangjin! Could you also commit this to branch-2.8? I find it odd that the fix is important enough to backport to 2.6 and 2.7 but not important enough for 2.8. Users updating from those older releases to 2.8 will accidentally drop this fix.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for pointing that out. I somehow (mistakenly) thought that 2.8.0 was already being cut. I just picked the fix to 2.8.0.

          Show
          sjlee0 Sangjin Lee added a comment - Thanks for pointing that out. I somehow (mistakenly) thought that 2.8.0 was already being cut. I just picked the fix to 2.8.0.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

            People

            • Assignee:
              sjlee0 Sangjin Lee
              Reporter:
              sjlee0 Sangjin Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development