Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3626

On Windows localized resources are not moved to the front of the classpath when they should be

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: yarn
    • Labels:
      None
    • Environment:

      Windows

    • Hadoop Flags:
      Reviewed

      Description

      In response to the mapreduce.job.user.classpath.first setting the classpath is ordered differently so that localized resources will appear before system classpath resources when tasks execute. On Windows this does not work because the localized resources are not linked into their final location when the classpath jar is created. To compensate for that localized jar resources are added directly to the classpath generated for the jar rather than being discovered from the localized directories. Unfortunately, they are always appended to the classpath, and so are never preferred over system resources.

      1. YARN-3626.0.patch
        6 kB
        Craig Welch
      2. YARN-3626.4.patch
        13 kB
        Craig Welch
      3. YARN-3626.6.patch
        13 kB
        Craig Welch
      4. YARN-3626.9.patch
        6 kB
        Craig Welch
      5. YARN-3626.11.patch
        7 kB
        Craig Welch
      6. YARN-3626.14.patch
        9 kB
        Craig Welch
      7. YARN-3626.15.patch
        9 kB
        Craig Welch
      8. YARN-3626.16.patch
        9 kB
        Craig Welch

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2157 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2157/)
          YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2157 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2157/ ) YARN-3626 . On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #209 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/209/)
          YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #209 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/209/ ) YARN-3626 . On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #199 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/199/)
          YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #199 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/199/ ) YARN-3626 . On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2139 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2139/)
          YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2139 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2139/ ) YARN-3626 . On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #941 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/941/)
          YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #941 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/941/ ) YARN-3626 . On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #211 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/211/)
          YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #211 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/211/ ) YARN-3626 . On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7909 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7909/)
          YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7909 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7909/ ) YARN-3626 . On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. (cnauroth: rev 4102e5882e17b75507ae5cf8b8979485b3e24cbc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          Hide
          cnauroth Chris Nauroth added a comment -

          I committed this to trunk, branch-2 and branch-2.7. Thanks for taking care of this, Craig!

          Show
          cnauroth Chris Nauroth added a comment - I committed this to trunk, branch-2 and branch-2.7. Thanks for taking care of this, Craig!
          Hide
          cwelch Craig Welch added a comment -

          The checkstyle is insignificant, the rest is all good.

          Show
          cwelch Craig Welch added a comment - The checkstyle is insignificant, the rest is all good.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Sure, annotated also

          Tx, that is what I meant..

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Sure, annotated also Tx, that is what I meant..
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 17s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 44s There were no new javac warning messages.
          +1 javadoc 10m 1s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 34s The applied patch generated 1 new checkstyle issues (total was 241, now 238).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
          +1 findbugs 4m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 9m 59s Tests passed in hadoop-mapreduce-client-app.
          +1 mapreduce tests 0m 45s Tests passed in hadoop-mapreduce-client-common.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 6m 10s Tests passed in hadoop-yarn-server-nodemanager.
              60m 8s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735424/YARN-3626.16.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 7dba700
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8090/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8090/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 17s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 44s There were no new javac warning messages. +1 javadoc 10m 1s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 34s The applied patch generated 1 new checkstyle issues (total was 241, now 238). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. +1 findbugs 4m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 9m 59s Tests passed in hadoop-mapreduce-client-app. +1 mapreduce tests 0m 45s Tests passed in hadoop-mapreduce-client-common. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 6m 10s Tests passed in hadoop-yarn-server-nodemanager.     60m 8s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735424/YARN-3626.16.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7dba700 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8090/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8090/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8090/console This message was automatically generated.
          Hide
          cwelch Craig Welch added a comment -

          Sure, annotated also

          Show
          cwelch Craig Welch added a comment - Sure, annotated also
          Hide
          cnauroth Chris Nauroth added a comment -

          Thanks, Craig! We could potentially stick the @Private annotation directly onto ApplicationConstants#CLASSPATH_PREPEND_DISTCACHE. I'll let Vinod chime in on whether or not this was the intent of his feedback.

          +1 from me, pending Jenkins run.

          Show
          cnauroth Chris Nauroth added a comment - Thanks, Craig! We could potentially stick the @Private annotation directly onto ApplicationConstants#CLASSPATH_PREPEND_DISTCACHE . I'll let Vinod chime in on whether or not this was the intent of his feedback. +1 from me, pending Jenkins run.
          Hide
          cwelch Craig Welch added a comment -

          Vinod Kumar Vavilapalli indicated it's private & Windows specific, all for doing something better overall long term aka YARN-3685

          Chris Nauroth switched to simpler valueOf

          Show
          cwelch Craig Welch added a comment - Vinod Kumar Vavilapalli indicated it's private & Windows specific, all for doing something better overall long term aka YARN-3685 Chris Nauroth switched to simpler valueOf
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Let's mark the new env CLASSPATH_PREPEND_DISTCACHE as @Private and add a note also in the code saying so - I don't see any use beyond what we are doing. That and I think we should clean up this entire mess via YARN-3685. Okay?

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Let's mark the new env CLASSPATH_PREPEND_DISTCACHE as @Private and add a note also in the code saying so - I don't see any use beyond what we are doing. That and I think we should clean up this entire mess via YARN-3685 . Okay?
          Hide
          cnauroth Chris Nauroth added a comment -

          Hi Craig. This looks good to me. I have just one minor nitpick. I think the logic in ContainerLaunch for setting preferLocalizedJars could be simplified to this:

          boolean preferLocalizedJars = Boolean.valueOf(classpathPrependDistCache);
          

          Boolean#valueOf is null-safe.

          Thanks!

          Show
          cnauroth Chris Nauroth added a comment - Hi Craig. This looks good to me. I have just one minor nitpick. I think the logic in ContainerLaunch for setting preferLocalizedJars could be simplified to this: boolean preferLocalizedJars = Boolean .valueOf(classpathPrependDistCache); Boolean#valueOf is null-safe. Thanks!
          Hide
          cwelch Craig Welch added a comment -

          Checkstyle looks insignificant.

          Chris Nauroth, Vinod Kumar Vavilapalli, I've changed the approach to use the environment instead of configuration as suggested, can one of you review pls?

          Show
          cwelch Craig Welch added a comment - Checkstyle looks insignificant. Chris Nauroth , Vinod Kumar Vavilapalli , I've changed the approach to use the environment instead of configuration as suggested, can one of you review pls?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 42s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 34s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 22s The applied patch generated 1 new checkstyle issues (total was 240, now 238).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 9m 26s Tests passed in hadoop-mapreduce-client-app.
          +1 mapreduce tests 0m 45s Tests passed in hadoop-mapreduce-client-common.
          +1 yarn tests 0m 30s Tests passed in hadoop-yarn-api.
          +1 yarn tests 6m 5s Tests passed in hadoop-yarn-server-nodemanager.
              58m 2s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12734054/YARN-3626.14.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / ce53c8e
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8015/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8015/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 42s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 22s The applied patch generated 1 new checkstyle issues (total was 240, now 238). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 9m 26s Tests passed in hadoop-mapreduce-client-app. +1 mapreduce tests 0m 45s Tests passed in hadoop-mapreduce-client-common. +1 yarn tests 0m 30s Tests passed in hadoop-yarn-api. +1 yarn tests 6m 5s Tests passed in hadoop-yarn-server-nodemanager.     58m 2s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12734054/YARN-3626.14.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ce53c8e checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8015/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8015/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8015/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 44s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 javadoc 9m 38s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 240, now 238).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
          +1 findbugs 3m 26s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 0m 46s Tests passed in hadoop-mapreduce-client-common.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api.
          +1 yarn tests 6m 1s Tests passed in hadoop-yarn-server-nodemanager.
              47m 15s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733294/YARN-3626.11.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / ac742c7
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7957/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7957/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7957/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7957/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7957/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7957/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 44s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 240, now 238). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 3m 26s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 0m 46s Tests passed in hadoop-mapreduce-client-common. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api. +1 yarn tests 6m 1s Tests passed in hadoop-yarn-server-nodemanager.     47m 15s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733294/YARN-3626.11.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ac742c7 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7957/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7957/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7957/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7957/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7957/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7957/console This message was automatically generated.
          Hide
          cwelch Craig Welch added a comment -

          Now using the environment to pass the configuration.

          Show
          cwelch Craig Welch added a comment - Now using the environment to pass the configuration.
          Hide
          cnauroth Chris Nauroth added a comment -

          I see now. Thanks for the clarification. In that case, I agree with the new proposal.

          Show
          cnauroth Chris Nauroth added a comment - I see now. Thanks for the clarification. In that case, I agree with the new proposal.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          I thought this implemented "per app" behavior, because it could vary between MR app submission instances. It would not be a requirement to put YARN_APPLICATION_CLASSPATH_PREPEND_DISTCACHE into the server configs and have the client and server share configs.

          YARN doesn't have a notion of app-configs, it doesn't know app's config files etc. So, the app cannot set a config property that it expects the server to respect.

          No idea, how your original patch apparently worked. May be we are missing something.

          Craig Welch, what I was proposing was something in the lines of (a) user sets MR user-classpath-before config (b) MR converts that into a special env for YARN and (c) YARN looks at the ENV to figure out how to the order the classpath.

          Overall, it is terrible that we are talking classpaths in YARN, but that's for another JIRA.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - I thought this implemented "per app" behavior, because it could vary between MR app submission instances. It would not be a requirement to put YARN_APPLICATION_CLASSPATH_PREPEND_DISTCACHE into the server configs and have the client and server share configs. YARN doesn't have a notion of app-configs, it doesn't know app's config files etc. So, the app cannot set a config property that it expects the server to respect. No idea, how your original patch apparently worked. May be we are missing something. Craig Welch , what I was proposing was something in the lines of (a) user sets MR user-classpath-before config (b) MR converts that into a special env for YARN and (c) YARN looks at the ENV to figure out how to the order the classpath. Overall, it is terrible that we are talking classpaths in YARN, but that's for another JIRA.
          Hide
          cwelch Craig Welch added a comment -

          Right, going back to Chris Nauroth, Vinod Kumar Vavilapalli, we chatted and you asserted that the original approach can't work, but it seemed to, it's not entirely clear to me why it shouldn't...

          Show
          cwelch Craig Welch added a comment - Right, going back to Chris Nauroth , Vinod Kumar Vavilapalli , we chatted and you asserted that the original approach can't work, but it seemed to, it's not entirely clear to me why it shouldn't...
          Hide
          cwelch Craig Welch added a comment -

          Can't we do the above? We definitely cannot insert mapreduce incantations like job.jar into YARN.

          That's why I took the config based approach, which apparently is invalid... but it also worked, which is quite confusing. I'm going to go back and validate our reasoning for believing it shoudn't.

          Can't we do the above? We definitely cannot insert mapreduce incantations like job.jar into YARN.

          I suppose we can if it would work. It needs to be something which can be propagated from Oozie, which adds additional complexity. Ideally, we need something the MrApps can set based on the presence of the mapred configuration so that it propagates through. Do we have an example of this being done elsewhere?

          Show
          cwelch Craig Welch added a comment - Can't we do the above? We definitely cannot insert mapreduce incantations like job.jar into YARN. That's why I took the config based approach, which apparently is invalid... but it also worked, which is quite confusing. I'm going to go back and validate our reasoning for believing it shoudn't. Can't we do the above? We definitely cannot insert mapreduce incantations like job.jar into YARN. I suppose we can if it would work. It needs to be something which can be propagated from Oozie, which adds additional complexity. Ideally, we need something the MrApps can set based on the presence of the mapred configuration so that it propagates through. Do we have an example of this being done elsewhere?
          Hide
          cnauroth Chris Nauroth added a comment -

          I don't fully understand the objection to the former patch that had been committed.

          The new configuration added is supposed to be per app, but it is now a server side configuration.

          There was a new YARN configuration property for triggering this behavior, but the MR application would toggle on that YARN property only if the MR job submission had MAPREDUCE_JOB_USER_CLASSPATH_FIRST on. From MRApps:

              boolean userClassesTakesPrecedence = 
                conf.getBoolean(MRJobConfig.MAPREDUCE_JOB_USER_CLASSPATH_FIRST, false);
          
              if (userClassesTakesPrecedence) {
                conf.set(YarnConfiguration.YARN_APPLICATION_CLASSPATH_PREPEND_DISTCACHE,
                  "true");
              }
          

          I thought this implemented "per app" behavior, because it could vary between MR app submission instances. It would not be a requirement to put YARN_APPLICATION_CLASSPATH_PREPEND_DISTCACHE into the server configs and have the client and server share configs.

          Is there a detail I'm missing?

          Show
          cnauroth Chris Nauroth added a comment - I don't fully understand the objection to the former patch that had been committed. The new configuration added is supposed to be per app, but it is now a server side configuration. There was a new YARN configuration property for triggering this behavior, but the MR application would toggle on that YARN property only if the MR job submission had MAPREDUCE_JOB_USER_CLASSPATH_FIRST on. From MRApps : boolean userClassesTakesPrecedence = conf.getBoolean(MRJobConfig.MAPREDUCE_JOB_USER_CLASSPATH_FIRST, false ); if (userClassesTakesPrecedence) { conf.set(YarnConfiguration.YARN_APPLICATION_CLASSPATH_PREPEND_DISTCACHE, " true " ); } I thought this implemented "per app" behavior, because it could vary between MR app submission instances. It would not be a requirement to put YARN_APPLICATION_CLASSPATH_PREPEND_DISTCACHE into the server configs and have the client and server share configs. Is there a detail I'm missing?
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          We may have to depend on some sort of a named environment variable or something, assuming adding a new field in CLC is not desirable.

          Can't we do the above? We definitely cannot insert mapreduce incantations like job.jar into YARN.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - We may have to depend on some sort of a named environment variable or something, assuming adding a new field in CLC is not desirable. Can't we do the above? We definitely cannot insert mapreduce incantations like job.jar into YARN.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 24s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 47s There were no new javac warning messages.
          +1 javadoc 10m 7s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 16s The applied patch generated 2 new checkstyle issues (total was 59, now 58).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 33s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 0m 45s Tests passed in hadoop-mapreduce-client-common.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 6m 1s Tests passed in hadoop-yarn-server-nodemanager.
              48m 49s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733257/YARN-3626.9.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f7e051c
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7954/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt
          hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7954/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7954/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7954/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7954/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7954/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 24s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 47s There were no new javac warning messages. +1 javadoc 10m 7s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 16s The applied patch generated 2 new checkstyle issues (total was 59, now 58). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 33s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 0m 45s Tests passed in hadoop-mapreduce-client-common. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 6m 1s Tests passed in hadoop-yarn-server-nodemanager.     48m 49s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733257/YARN-3626.9.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f7e051c checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7954/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7954/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7954/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7954/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7954/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7954/console This message was automatically generated.
          Hide
          cwelch Craig Welch added a comment -

          Chris Nauroth, Varun Vasudev - This patch goes back to the original approach I passed by you offline - the fix itself is the same, but it uses the classpath instead of configuration to determine when the behavior should change. Your thoughts?

          Show
          cwelch Craig Welch added a comment - Chris Nauroth , Varun Vasudev - This patch goes back to the original approach I passed by you offline - the fix itself is the same, but it uses the classpath instead of configuration to determine when the behavior should change. Your thoughts?
          Hide
          cwelch Craig Welch added a comment -

          In that case, here's a patch which goes back to the original approach used during troubleshooting, which uses the classpath itself to communicate the difference (it only touches other code to revert parts of the earlier patch no longer needed, the actual change, when done this way, is solely in ContainerLaunch.java, and it makes the conditional determination based on the classpath differences already present due to the manipulation earlier in the chain, in this case, by mapreduce due to user.classpath.first)

          Show
          cwelch Craig Welch added a comment - In that case, here's a patch which goes back to the original approach used during troubleshooting, which uses the classpath itself to communicate the difference (it only touches other code to revert parts of the earlier patch no longer needed, the actual change, when done this way, is solely in ContainerLaunch.java, and it makes the conditional determination based on the classpath differences already present due to the manipulation earlier in the chain, in this case, by mapreduce due to user.classpath.first)
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          As I understand, the current patch doesn't do what you intend. The new configuration added is supposed to be per app, but it is now a server side configuration. Unless the app and server share configurations, this should not work.

          Reopening this. I think we should revert it (/cc Xuan Gong) and continue discussion/work to do it the right away.

          We may have to depend on some sort of a named environment variable or something, assuming adding a new field in CLC is not desirable.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - As I understand, the current patch doesn't do what you intend. The new configuration added is supposed to be per app, but it is now a server side configuration. Unless the app and server share configurations, this should not work. Reopening this. I think we should revert it (/cc Xuan Gong ) and continue discussion/work to do it the right away. We may have to depend on some sort of a named environment variable or something, assuming adding a new field in CLC is not desirable.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2143 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2143/)
          YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2143 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2143/ ) YARN-3626 . On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #185 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/185/)
          YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #185 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/185/ ) YARN-3626 . On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2125 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2125/)
          YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2125 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2125/ ) YARN-3626 . On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #195 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/195/)
          YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #195 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/195/ ) YARN-3626 . On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #196 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/196/)
          YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #196 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/196/ ) YARN-3626 . On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #927 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/927/)
          YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #927 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/927/ ) YARN-3626 . On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-yarn-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7819 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7819/)
          YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7819 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7819/ ) YARN-3626 . On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch (xgong: rev 0f95921447ea547bdf9caf18f7fde46bc66031f8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          Hide
          xgong Xuan Gong added a comment -

          Committed into trunk/branch-2/branch-2.7. Thanks, Craig.

          Show
          xgong Xuan Gong added a comment - Committed into trunk/branch-2/branch-2.7. Thanks, Craig.
          Hide
          xgong Xuan Gong added a comment -

          +1 LGTM. Will commit

          Show
          xgong Xuan Gong added a comment - +1 LGTM. Will commit
          Hide
          cwelch Craig Welch added a comment -

          Checkstyle looks unimportant, everything else OK

          Show
          cwelch Craig Welch added a comment - Checkstyle looks unimportant, everything else OK
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 44s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 46s The applied patch generated 3 new checkstyle issues (total was 211, now 214).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 23s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 0m 46s Tests passed in hadoop-mapreduce-client-common.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api.
          +1 yarn tests 5m 59s Tests passed in hadoop-yarn-server-nodemanager.
              46m 57s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732645/YARN-3626.6.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / cdec12d
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7924/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7924/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7924/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7924/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7924/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7924/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 44s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 46s The applied patch generated 3 new checkstyle issues (total was 211, now 214). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 23s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 0m 46s Tests passed in hadoop-mapreduce-client-common. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api. +1 yarn tests 5m 59s Tests passed in hadoop-yarn-server-nodemanager.     46m 57s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732645/YARN-3626.6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / cdec12d checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7924/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7924/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7924/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7924/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7924/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7924/console This message was automatically generated.
          Hide
          cwelch Craig Welch added a comment -

          Fix broken unit tests

          Show
          cwelch Craig Welch added a comment - Fix broken unit tests
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 8s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 42s There were no new javac warning messages.
          +1 javadoc 9m 44s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 47s The applied patch generated 3 new checkstyle issues (total was 211, now 214).
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 27s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 0m 47s Tests passed in hadoop-mapreduce-client-common.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          -1 yarn tests 5m 11s Tests failed in hadoop-yarn-server-nodemanager.
              46m 54s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
            hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
            hadoop.yarn.server.nodemanager.containermanager.TestNMProxy
            hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
            hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732495/YARN-3626.4.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / fcd0702
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7912/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7912/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 8s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 42s There were no new javac warning messages. +1 javadoc 9m 44s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 47s The applied patch generated 3 new checkstyle issues (total was 211, now 214). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 27s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 0m 47s Tests passed in hadoop-mapreduce-client-common. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. -1 yarn tests 5m 11s Tests failed in hadoop-yarn-server-nodemanager.     46m 54s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManager   hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService   hadoop.yarn.server.nodemanager.containermanager.TestNMProxy   hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch   hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732495/YARN-3626.4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fcd0702 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7912/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7912/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7912/console This message was automatically generated.
          Hide
          cwelch Craig Welch added a comment -

          With testing

          Show
          cwelch Craig Welch added a comment - With testing
          Hide
          cwelch Craig Welch added a comment -

          The attached patch propagates the conditional as a yarn configuration option and moves localized resources to the front of the classpath when appropriate

          Show
          cwelch Craig Welch added a comment - The attached patch propagates the conditional as a yarn configuration option and moves localized resources to the front of the classpath when appropriate
          Hide
          cwelch Craig Welch added a comment -

          To resolve this, the situation should be detected and, when applicable, localized resources should be put at the beginning of the classpath rather than the end.

          Show
          cwelch Craig Welch added a comment - To resolve this, the situation should be detected and, when applicable, localized resources should be put at the beginning of the classpath rather than the end.

            People

            • Assignee:
              cwelch Craig Welch
              Reporter:
              cwelch Craig Welch
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development