Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: nodemanager
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Sometimes when a container fails, it can be pretty hard to figure out why it failed.

      My proposal is that if a container fails, we collect information about the container local dir and dump it into the container log dir. Ideally, I'd like to tar up the directory entirely, but I'm not sure of the security and space implications of such a approach. At the very least, we can list all the files in the container local dir, and dump the contents of launch_container.sh(into the container log dir).

      When log aggregation occurs, all this information will automatically get collected and make debugging such failures much easier.

      1. YARN-4309.010.patch
        20 kB
        Varun Vasudev
      2. YARN-4309.009.patch
        20 kB
        Varun Vasudev
      3. YARN-4309.008.patch
        19 kB
        Varun Vasudev
      4. YARN-4309.007.patch
        19 kB
        Varun Vasudev
      5. YARN-4309.006.patch
        19 kB
        Varun Vasudev
      6. YARN-4309.005.patch
        19 kB
        Varun Vasudev
      7. YARN-4309.004.patch
        18 kB
        Varun Vasudev
      8. YARN-4309.003.patch
        18 kB
        Varun Vasudev
      9. YARN-4309.002.patch
        17 kB
        Varun Vasudev
      10. YARN-4309.001.patch
        15 kB
        Varun Vasudev

        Issue Links

          Activity

          Hide
          aw Allen Wittenauer added a comment -

          We had a user who accidentally dist cached one of the base ETL data dirs... about 60GB worth... and then the job would fail. So now you're looking at 60GB * Y containers of data to collect before even getting to the other stuff that's relevant.

          Show
          aw Allen Wittenauer added a comment - We had a user who accidentally dist cached one of the base ETL data dirs... about 60GB worth... and then the job would fail. So now you're looking at 60GB * Y containers of data to collect before even getting to the other stuff that's relevant.
          Hide
          vvasudev Varun Vasudev added a comment -

          Yeah, tar-ing up the local dir may not be the best idea. However dumping the directory contents(file listings) and the contents of launch_container.sh should be ok?

          Show
          vvasudev Varun Vasudev added a comment - Yeah, tar-ing up the local dir may not be the best idea. However dumping the directory contents(file listings) and the contents of launch_container.sh should be ok?
          Hide
          aw Allen Wittenauer added a comment -

          As long as JH and/or TS respects the job ACLs when it comes to sharing the data and sets permissions on the files in the container log appropriately, nothing really sticks out. It's definitely private-to-the-user data though and full of information leaks.

          Show
          aw Allen Wittenauer added a comment - As long as JH and/or TS respects the job ACLs when it comes to sharing the data and sets permissions on the files in the container log appropriately, nothing really sticks out. It's definitely private-to-the-user data though and full of information leaks.
          Hide
          jlowe Jason Lowe added a comment -

          It's definitely private-to-the-user data though and full of information leaks.

          Yeah, agree with Allen that it's dicey to publish everything there. For example, the security tokens for the container are stored in one of the local container files, and we do not want that stored in HDFS and accessible by the jobhistoryserver user nor the ATS user. The nodemanager goes out of its way, via the container-executor, to make sure user-private files are not visible even to the nodemanager user.

          The launch script should be OK and is really the most valuable thing there for debugging startup failures. Almost everything in that script is derived from what's in the configs, and the configs are already stored in HDFS or the ATS.

          Show
          jlowe Jason Lowe added a comment - It's definitely private-to-the-user data though and full of information leaks. Yeah, agree with Allen that it's dicey to publish everything there. For example, the security tokens for the container are stored in one of the local container files, and we do not want that stored in HDFS and accessible by the jobhistoryserver user nor the ATS user. The nodemanager goes out of its way, via the container-executor, to make sure user-private files are not visible even to the nodemanager user. The launch script should be OK and is really the most valuable thing there for debugging startup failures. Almost everything in that script is derived from what's in the configs, and the configs are already stored in HDFS or the ATS.
          Hide
          kasha Karthik Kambatla added a comment -

          Generally, in favor of collecting debug information. launch_container.sh and directory listings/sizes should go a long way in our ability to debug launch failures. May be, collecting logs is okay for non-launch failures especially if the logs are not aggregated for some reason; would be nice to specify a limit on the size of these logs collected.

          Show
          kasha Karthik Kambatla added a comment - Generally, in favor of collecting debug information. launch_container.sh and directory listings/sizes should go a long way in our ability to debug launch failures. May be, collecting logs is okay for non-launch failures especially if the logs are not aggregated for some reason; would be nice to specify a limit on the size of these logs collected.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1 for ls, -1 for contents. That can include keytabs.

          Anything for initial launch would be invaluable, not just scripts, but cluster environment. Even there though, I'd like yarn-site.xml to support a list of env vars to strip, so no EC2 variables would leak out

          Show
          stevel@apache.org Steve Loughran added a comment - +1 for ls, -1 for contents. That can include keytabs. Anything for initial launch would be invaluable, not just scripts, but cluster environment. Even there though, I'd like yarn-site.xml to support a list of env vars to strip, so no EC2 variables would leak out
          Hide
          vvasudev Varun Vasudev added a comment -

          Steve Loughran - can you give me an example of what you mean by cluster environment?

          Show
          vvasudev Varun Vasudev added a comment - Steve Loughran - can you give me an example of what you mean by cluster environment?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I mean all the environment variables of the container

          Show
          stevel@apache.org Steve Loughran added a comment - I mean all the environment variables of the container
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded an initial version of the patch. It's a little difficult to collect the information only for failures and easier to collect it for all runs. Essentially, collecting the information for failures in secure mode is a lot harder and requires changes to container-executor. I've made generation of the additional debug information optional, with the default set to false.

          The patch creates a copy of launch_container.sh, the output of ls and the output of "find -L . -maxdepth 5 -ls".

          There's no particular reason for maxdepth 5 - I'm happy to change it if someone feels some other value is more appropriate. The reason for find and ls is that ls will output the symlinks whereas find gives you the size of the file pointed to by the symlink.

          This version of the patch is for Linux only. If someone knows the changes for Windows, I'll add those in.

          Just for information, for a mapreduce pi job, this is what was generated for the directory contents:

          ls:
          total 32
          -rw-r--r-- 1 varun varun  129 Nov 26 19:47 container_tokens
          -rwx------ 1 varun varun  702 Nov 26 19:47 default_container_executor_session.sh
          -rwx------ 1 varun varun  756 Nov 26 19:47 default_container_executor.sh
          lrwxrwxrwx 1 varun varun  113 Nov 26 19:47 job.jar -> /var/hadoop/hadoop-3-data/grid/local/usercache/varun/appcache/application_1448547413698_0001/filecache/10/job.jar
          lrwxrwxrwx 1 varun varun  114 Nov 26 19:47 job.xml -> /var/hadoop/hadoop-3-data/grid2/local/usercache/varun/appcache/application_1448547413698_0001/filecache/13/job.xml
          -rwx------ 1 varun varun 4941 Nov 26 19:47 launch_container.sh
          drwx--x--- 2 varun varun 4096 Nov 26 19:47 tmp
          find:
          1079692    4 drwx--x---   3 varun    varun        4096 Nov 26 19:47 .
          1074586    4 -rw-r--r--   1 varun    varun          16 Nov 26 19:47 ./.default_container_executor.sh.crc
          1074581    8 -rwx------   1 varun    varun        4941 Nov 26 19:47 ./launch_container.sh
          1049070  104 -r-x------   1 varun    varun      105105 Nov 26 19:47 ./job.xml
          1873872    4 drwx------   2 varun    varun        4096 Nov 26 19:47 ./job.jar
          1873870  272 -r-x------   1 varun    varun      275886 Nov 26 19:47 ./job.jar/job.jar
          1079695    4 drwx--x---   2 varun    varun        4096 Nov 26 19:47 ./tmp
          1074582    4 -rw-r--r--   1 varun    varun          48 Nov 26 19:47 ./.launch_container.sh.crc
          1074580    4 -rw-r--r--   1 varun    varun          12 Nov 26 19:47 ./.container_tokens.crc
          1074585    4 -rwx------   1 varun    varun         756 Nov 26 19:47 ./default_container_executor.sh
          1074583    4 -rwx------   1 varun    varun         702 Nov 26 19:47 ./default_container_executor_session.sh
          1074579    4 -rw-r--r--   1 varun    varun         129 Nov 26 19:47 ./container_tokens
          1074584    4 -rw-r--r--   1 varun    varun          16 Nov 26 19:47 ./.default_container_executor_session.sh.crc
          
          Show
          vvasudev Varun Vasudev added a comment - Uploaded an initial version of the patch. It's a little difficult to collect the information only for failures and easier to collect it for all runs. Essentially, collecting the information for failures in secure mode is a lot harder and requires changes to container-executor. I've made generation of the additional debug information optional, with the default set to false. The patch creates a copy of launch_container.sh, the output of ls and the output of "find -L . -maxdepth 5 -ls". There's no particular reason for maxdepth 5 - I'm happy to change it if someone feels some other value is more appropriate. The reason for find and ls is that ls will output the symlinks whereas find gives you the size of the file pointed to by the symlink. This version of the patch is for Linux only. If someone knows the changes for Windows, I'll add those in. Just for information, for a mapreduce pi job, this is what was generated for the directory contents: ls: total 32 -rw-r--r-- 1 varun varun 129 Nov 26 19:47 container_tokens -rwx------ 1 varun varun 702 Nov 26 19:47 default_container_executor_session.sh -rwx------ 1 varun varun 756 Nov 26 19:47 default_container_executor.sh lrwxrwxrwx 1 varun varun 113 Nov 26 19:47 job.jar -> / var /hadoop/hadoop-3-data/grid/local/usercache/varun/appcache/application_1448547413698_0001/filecache/10/job.jar lrwxrwxrwx 1 varun varun 114 Nov 26 19:47 job.xml -> / var /hadoop/hadoop-3-data/grid2/local/usercache/varun/appcache/application_1448547413698_0001/filecache/13/job.xml -rwx------ 1 varun varun 4941 Nov 26 19:47 launch_container.sh drwx--x--- 2 varun varun 4096 Nov 26 19:47 tmp find: 1079692 4 drwx--x--- 3 varun varun 4096 Nov 26 19:47 . 1074586 4 -rw-r--r-- 1 varun varun 16 Nov 26 19:47 ./.default_container_executor.sh.crc 1074581 8 -rwx------ 1 varun varun 4941 Nov 26 19:47 ./launch_container.sh 1049070 104 -r-x------ 1 varun varun 105105 Nov 26 19:47 ./job.xml 1873872 4 drwx------ 2 varun varun 4096 Nov 26 19:47 ./job.jar 1873870 272 -r-x------ 1 varun varun 275886 Nov 26 19:47 ./job.jar/job.jar 1079695 4 drwx--x--- 2 varun varun 4096 Nov 26 19:47 ./tmp 1074582 4 -rw-r--r-- 1 varun varun 48 Nov 26 19:47 ./.launch_container.sh.crc 1074580 4 -rw-r--r-- 1 varun varun 12 Nov 26 19:47 ./.container_tokens.crc 1074585 4 -rwx------ 1 varun varun 756 Nov 26 19:47 ./default_container_executor.sh 1074583 4 -rwx------ 1 varun varun 702 Nov 26 19:47 ./default_container_executor_session.sh 1074579 4 -rw-r--r-- 1 varun varun 129 Nov 26 19:47 ./container_tokens 1074584 4 -rw-r--r-- 1 varun varun 16 Nov 26 19:47 ./.default_container_executor_session.sh.crc
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 20s trunk passed
          +1 compile 2m 14s trunk passed with JDK v1.8.0_66
          +1 compile 2m 25s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 36s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 4m 7s trunk passed
          +1 javadoc 1m 28s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 49s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 27s the patch passed
          +1 compile 2m 0s the patch passed with JDK v1.8.0_66
          -1 javac 9m 44s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66 generated 5 new issues (was 14, now 14).
          +1 javac 2m 0s the patch passed
          +1 compile 2m 16s the patch passed with JDK v1.7.0_85
          -1 javac 12m 0s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_85 with JDK v1.7.0_85 generated 5 new issues (was 15, now 15).
          +1 javac 2m 16s the patch passed
          -1 checkstyle 0m 29s Patch generated 5 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 356, now 359).
          +1 mvnsite 1m 32s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 4m 10s the patch passed
          +1 javadoc 1m 28s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 51s the patch passed with JDK v1.7.0_85
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 58s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          -1 unit 8m 53s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          +1 unit 2m 12s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
          -1 unit 9m 19s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_85.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          68m 12s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
          JDK v1.7.0_85 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774533/YARN-4309.001.patch
          JIRA Issue YARN-4309
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 70608490ced0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9f256d1
          findbugs v3.0.0
          javac hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/9804/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt
          javac hadoop-yarn-project_hadoop-yarn-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-YARN-Build/9804/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_85.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9804/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9804/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9804/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9804/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9804/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9804/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9804/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 20s trunk passed +1 compile 2m 14s trunk passed with JDK v1.8.0_66 +1 compile 2m 25s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 36s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 4m 7s trunk passed +1 javadoc 1m 28s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 49s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 27s the patch passed +1 compile 2m 0s the patch passed with JDK v1.8.0_66 -1 javac 9m 44s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66 generated 5 new issues (was 14, now 14). +1 javac 2m 0s the patch passed +1 compile 2m 16s the patch passed with JDK v1.7.0_85 -1 javac 12m 0s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_85 with JDK v1.7.0_85 generated 5 new issues (was 15, now 15). +1 javac 2m 16s the patch passed -1 checkstyle 0m 29s Patch generated 5 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 356, now 359). +1 mvnsite 1m 32s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 4m 10s the patch passed +1 javadoc 1m 28s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 51s the patch passed with JDK v1.7.0_85 +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 1m 58s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. -1 unit 8m 53s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. +1 unit 2m 12s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. -1 unit 9m 19s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_85. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 68m 12s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch JDK v1.7.0_85 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774533/YARN-4309.001.patch JIRA Issue YARN-4309 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 70608490ced0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9f256d1 findbugs v3.0.0 javac hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/9804/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt javac hadoop-yarn-project_hadoop-yarn-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-YARN-Build/9804/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_85.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9804/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9804/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9804/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9804/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9804/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9804/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9804/console This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          This version of the patch is for Linux only.

          Linux or Unix?

          Show
          aw Allen Wittenauer added a comment - This version of the patch is for Linux only. Linux or Unix?
          Hide
          vvasudev Varun Vasudev added a comment -

          bq .Linux or Unix?
          Sorry, not Linux - this patch is for Unix only.

          Uploaded a new version of the patch to address the issues raised by the pre-commit build.

          Show
          vvasudev Varun Vasudev added a comment - bq .Linux or Unix? Sorry, not Linux - this patch is for Unix only. Uploaded a new version of the patch to address the issues raised by the pre-commit build.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 58s trunk passed
          +1 compile 1m 59s trunk passed with JDK v1.8.0_66
          +1 compile 2m 15s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 31s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 3m 45s trunk passed
          +1 javadoc 1m 27s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 46s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 25s the patch passed
          +1 compile 1m 58s the patch passed with JDK v1.8.0_66
          +1 javac 1m 58s the patch passed
          +1 compile 2m 15s the patch passed with JDK v1.7.0_85
          +1 javac 2m 15s the patch passed
          -1 checkstyle 0m 28s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 357, now 356).
          +1 mvnsite 1m 31s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 4m 11s the patch passed
          +1 javadoc 1m 28s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 45s the patch passed with JDK v1.7.0_85
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 58s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 44s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          +1 unit 2m 13s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
          +1 unit 9m 11s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          66m 20s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774587/YARN-4309.002.patch
          JIRA Issue YARN-4309
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux c3587662ea1c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9f256d1
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9807/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9807/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 75MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9807/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 58s trunk passed +1 compile 1m 59s trunk passed with JDK v1.8.0_66 +1 compile 2m 15s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 31s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 3m 45s trunk passed +1 javadoc 1m 27s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 46s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 25s the patch passed +1 compile 1m 58s the patch passed with JDK v1.8.0_66 +1 javac 1m 58s the patch passed +1 compile 2m 15s the patch passed with JDK v1.7.0_85 +1 javac 2m 15s the patch passed -1 checkstyle 0m 28s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 357, now 356). +1 mvnsite 1m 31s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 4m 11s the patch passed +1 javadoc 1m 28s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 45s the patch passed with JDK v1.7.0_85 +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 1m 58s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 44s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. +1 unit 2m 13s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. +1 unit 9m 11s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 66m 20s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774587/YARN-4309.002.patch JIRA Issue YARN-4309 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux c3587662ea1c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9f256d1 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9807/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9807/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9807/console This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new patch that adds a section on broken symlinks to the directory info.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new patch that adds a section on broken symlinks to the directory info.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 9m 12s trunk passed
          +1 compile 2m 37s trunk passed with JDK v1.8.0_66
          +1 compile 2m 13s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 33s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 3m 56s trunk passed
          +1 javadoc 1m 33s trunk passed with JDK v1.8.0_66
          +1 javadoc 4m 7s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 39s the patch passed
          +1 compile 2m 31s the patch passed with JDK v1.8.0_66
          +1 javac 2m 31s the patch passed
          +1 compile 2m 43s the patch passed with JDK v1.7.0_85
          +1 javac 2m 43s the patch passed
          -1 checkstyle 0m 35s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 357, now 356).
          +1 mvnsite 1m 48s the patch passed
          +1 mvneclipse 0m 43s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 4m 25s the patch passed
          +1 javadoc 1m 31s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 46s the patch passed with JDK v1.7.0_85
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 58s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 49s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 28s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          +1 unit 2m 23s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
          +1 unit 9m 11s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          71m 14s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774974/YARN-4309.003.patch
          JIRA Issue YARN-4309
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 2b4d4459eed9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1c05393
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9822/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9822/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9822/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 12s trunk passed +1 compile 2m 37s trunk passed with JDK v1.8.0_66 +1 compile 2m 13s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 33s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 3m 56s trunk passed +1 javadoc 1m 33s trunk passed with JDK v1.8.0_66 +1 javadoc 4m 7s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 39s the patch passed +1 compile 2m 31s the patch passed with JDK v1.8.0_66 +1 javac 2m 31s the patch passed +1 compile 2m 43s the patch passed with JDK v1.7.0_85 +1 javac 2m 43s the patch passed -1 checkstyle 0m 35s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 357, now 356). +1 mvnsite 1m 48s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 4m 25s the patch passed +1 javadoc 1m 31s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 46s the patch passed with JDK v1.7.0_85 +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 1m 58s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 49s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 28s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. +1 unit 2m 23s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. +1 unit 9m 11s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 71m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774974/YARN-4309.003.patch JIRA Issue YARN-4309 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 2b4d4459eed9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1c05393 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9822/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9822/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9822/console This message was automatically generated.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Varun Vasudev , I took a look at the patch. Couple of comments :

          • Could you clarify why the debugging information gathering in DockerContainerExecutor.writeLaunchEnv is not guarded by a config check? The new test you added uses DefaultContainerExecutor so it looks like this was missed.
          • There seem to be minor inconsistent line spacing issues in the new test function in TestContainerLaunch.java

          Apart from these, assuming it is safe to list user directory contents (as already discussed on this JIRA), the patch seems good to me. Thanks for this patch - I expect the launch_container.sh copy to be particularly useful for debugging purposes.

          Show
          sidharta-s Sidharta Seethana added a comment - Varun Vasudev , I took a look at the patch. Couple of comments : Could you clarify why the debugging information gathering in DockerContainerExecutor.writeLaunchEnv is not guarded by a config check? The new test you added uses DefaultContainerExecutor so it looks like this was missed. There seem to be minor inconsistent line spacing issues in the new test function in TestContainerLaunch.java Apart from these, assuming it is safe to list user directory contents (as already discussed on this JIRA), the patch seems good to me. Thanks for this patch - I expect the launch_container.sh copy to be particularly useful for debugging purposes.
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the review Sidharta Seethana.

          Could you clarify why the debugging information gathering in DockerContainerExecutor.writeLaunchEnv is not guarded by a config check?

          Good catch. The config check should be present in DockerContainerExecutor as well. Fixed.

          There seem to be minor inconsistent line spacing issues in the new test function in TestContainerLaunch.java

          Fixed.

          I've changed the find command in the latest version to not use the xtype option which seems to be Linux only. I've also renamed the scriptbuilder functions to indicate that they're meant for debugging purposes.

          Show
          vvasudev Varun Vasudev added a comment - Thanks for the review Sidharta Seethana . Could you clarify why the debugging information gathering in DockerContainerExecutor.writeLaunchEnv is not guarded by a config check? Good catch. The config check should be present in DockerContainerExecutor as well. Fixed. There seem to be minor inconsistent line spacing issues in the new test function in TestContainerLaunch.java Fixed. I've changed the find command in the latest version to not use the xtype option which seems to be Linux only. I've also renamed the scriptbuilder functions to indicate that they're meant for debugging purposes.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 28s trunk passed
          +1 compile 2m 1s trunk passed with JDK v1.8.0_66
          +1 compile 2m 16s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 33s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 3m 44s trunk passed
          +1 javadoc 1m 29s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 46s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 26s the patch passed
          +1 compile 2m 0s the patch passed with JDK v1.8.0_66
          +1 javac 2m 0s the patch passed
          +1 compile 2m 21s the patch passed with JDK v1.7.0_85
          +1 javac 2m 21s the patch passed
          -1 checkstyle 0m 30s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 356, now 356).
          +1 mvnsite 1m 33s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 4m 25s the patch passed
          +1 javadoc 1m 30s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 50s the patch passed with JDK v1.7.0_85
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 59s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 56s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          +1 unit 2m 16s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
          +1 unit 9m 22s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          67m 58s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775246/YARN-4309.004.patch
          JIRA Issue YARN-4309
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 95cbd814b33b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 53e3bf7
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9835/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9835/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9835/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 28s trunk passed +1 compile 2m 1s trunk passed with JDK v1.8.0_66 +1 compile 2m 16s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 33s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 3m 44s trunk passed +1 javadoc 1m 29s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 46s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 26s the patch passed +1 compile 2m 0s the patch passed with JDK v1.8.0_66 +1 javac 2m 0s the patch passed +1 compile 2m 21s the patch passed with JDK v1.7.0_85 +1 javac 2m 21s the patch passed -1 checkstyle 0m 30s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 356, now 356). +1 mvnsite 1m 33s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 4m 25s the patch passed +1 javadoc 1m 30s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 50s the patch passed with JDK v1.7.0_85 +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 1m 59s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 56s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. +1 unit 2m 16s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. +1 unit 9m 22s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 67m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775246/YARN-4309.004.patch JIRA Issue YARN-4309 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 95cbd814b33b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 53e3bf7 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9835/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9835/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9835/console This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new version of the patch with Windows support.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new version of the patch with Windows support.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 44s trunk passed
          +1 compile 2m 38s trunk passed with JDK v1.8.0_66
          +1 compile 2m 24s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 37s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 4m 5s trunk passed
          +1 javadoc 1m 35s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 52s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 30s the patch passed
          +1 compile 2m 17s the patch passed with JDK v1.8.0_66
          +1 javac 2m 17s the patch passed
          +1 compile 2m 24s the patch passed with JDK v1.7.0_85
          +1 javac 2m 24s the patch passed
          -1 checkstyle 0m 30s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 357, now 358).
          +1 mvnsite 1m 39s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 4m 29s the patch passed
          +1 javadoc 1m 40s the patch passed with JDK v1.8.0_66
          +1 javadoc 4m 3s the patch passed with JDK v1.7.0_85
          +1 unit 0m 29s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 10s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 9m 2s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 28s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          +1 unit 2m 24s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
          +1 unit 9m 23s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          71m 14s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775317/YARN-4309.005.patch
          JIRA Issue YARN-4309
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux a428790d487f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 53e3bf7
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9838/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9838/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9838/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 44s trunk passed +1 compile 2m 38s trunk passed with JDK v1.8.0_66 +1 compile 2m 24s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 37s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 4m 5s trunk passed +1 javadoc 1m 35s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 52s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 30s the patch passed +1 compile 2m 17s the patch passed with JDK v1.8.0_66 +1 javac 2m 17s the patch passed +1 compile 2m 24s the patch passed with JDK v1.7.0_85 +1 javac 2m 24s the patch passed -1 checkstyle 0m 30s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 357, now 358). +1 mvnsite 1m 39s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 4m 29s the patch passed +1 javadoc 1m 40s the patch passed with JDK v1.8.0_66 +1 javadoc 4m 3s the patch passed with JDK v1.7.0_85 +1 unit 0m 29s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 10s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 9m 2s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 28s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. +1 unit 2m 24s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. +1 unit 9m 23s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 71m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775317/YARN-4309.005.patch JIRA Issue YARN-4309 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux a428790d487f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 53e3bf7 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9838/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9838/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9838/console This message was automatically generated.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          hi Varun Vasudev,

          I am using the find command that you have in the patch against broken symlinks - it is not clear to me how broken symlink info is captured (please see below). Could you please clarify?

          q (19:50:35) ~/symlink-test$ ls -l
          total 0
          q (19:50:47) ~/symlink-test$ ln -s world hello
          q (19:51:03) ~/symlink-test$ find -L . -maxdepth 5 -type l -ls
          2149279432    0 lrwxrwxrwx   1 sseethana sseethana        5 Dec  2 19:51 ./hello -> world
          q (19:51:15) ~/symlink-test$ echo $?
          0
          q (19:51:52) ~/symlink-test$ uname -a
          Linux q 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
          
          Show
          sidharta-s Sidharta Seethana added a comment - hi Varun Vasudev , I am using the find command that you have in the patch against broken symlinks - it is not clear to me how broken symlink info is captured (please see below). Could you please clarify? q (19:50:35) ~/symlink-test$ ls -l total 0 q (19:50:47) ~/symlink-test$ ln -s world hello q (19:51:03) ~/symlink-test$ find -L . -maxdepth 5 -type l -ls 2149279432 0 lrwxrwxrwx 1 sseethana sseethana 5 Dec 2 19:51 ./hello -> world q (19:51:15) ~/symlink-test$ echo $? 0 q (19:51:52) ~/symlink-test$ uname -a Linux q 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Ah, I guess the fact that it is listed implies the link is broken. I was thrown off by this comment in the patch which seemed to imply find would exit with an error of some sort.

                // don't run error check because if there are broken symlinks
                // or loops find will exit with an error
          

          Also, do we need to worry about -L following links outside of the current directory?

          Show
          sidharta-s Sidharta Seethana added a comment - Ah, I guess the fact that it is listed implies the link is broken. I was thrown off by this comment in the patch which seemed to imply find would exit with an error of some sort. // don't run error check because if there are broken symlinks // or loops find will exit with an error Also, do we need to worry about -L following links outside of the current directory?
          Hide
          vvasudev Varun Vasudev added a comment -

          do we need to worry about -L following links outside of the current directory?

          find will follow the links outside the current directory upto the maxdepth. This is useful to because we symlink to resources outside the work dir from the container work dir(like the mapreduce jar, the job conf, etc).

          Show
          vvasudev Varun Vasudev added a comment - do we need to worry about -L following links outside of the current directory? find will follow the links outside the current directory upto the maxdepth. This is useful to because we symlink to resources outside the work dir from the container work dir(like the mapreduce jar, the job conf, etc).
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Could you please edit the comment mentioned above to make it a bit clearer? Maybe include a note, both in yarn-default.xml 's description of the config flag as well as code that symlinks could be followed outside the current directory?

          Also, there seem to be a few spurious empty lines introduced in DockerContainerExecutor. Apart from this, the latest patch seems good to me.

          Show
          sidharta-s Sidharta Seethana added a comment - Could you please edit the comment mentioned above to make it a bit clearer? Maybe include a note, both in yarn-default.xml 's description of the config flag as well as code that symlinks could be followed outside the current directory? Also, there seem to be a few spurious empty lines introduced in DockerContainerExecutor. Apart from this, the latest patch seems good to me.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Hi Varun Vasudev,

          Thanks for working on this task, it's really useful to identify container launch issues, some questions/comments:

          • Since debug information fetch script (like copy script and list files) is at the end of launch_container.sh, is it possible that a container is killed so such script cannot be executed?
          • Do you think is it better to generate a separated script file to fetch debug information before launch user code? Which we can
            1. Guarantee it will be executed
            2. It won't add debug information to normal launch_container.sh.
            3. Return code of script won't affected by debug script.
          • Is it possible to enable/disable this function while NM is running?

          +Sidharta Seethana.

          Show
          leftnoteasy Wangda Tan added a comment - Hi Varun Vasudev , Thanks for working on this task, it's really useful to identify container launch issues, some questions/comments: Since debug information fetch script (like copy script and list files) is at the end of launch_container.sh, is it possible that a container is killed so such script cannot be executed? Do you think is it better to generate a separated script file to fetch debug information before launch user code? Which we can 1. Guarantee it will be executed 2. It won't add debug information to normal launch_container.sh. 3. Return code of script won't affected by debug script. Is it possible to enable/disable this function while NM is running? + Sidharta Seethana .
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new patch to address Sidharta Seethana's comments.

          Wangda Tan -

          Since debug information fetch script (like copy script and list files) is at the end of launch_container.sh, is it possible that a container is killed so such script cannot be executed?

          It's not at the end - it's just before the actually container process is launched so if we reach a stage where we are ready to call launch_container.sh it should almost always be run. This is what the relevant lines from launch_container.sh look like with the patch:

          echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/var/hadoop/hadoop-3-data/grid/log/application_1449046677123_0002/container_1449046677123_0002_01_000001/directory.info"
          find -L . -maxdepth 5 -type l -ls 1>>"/var/hadoop/hadoop-3-data/grid/log/application_1449046677123_0002/container_1449046677123_0002_01_000001/directory.info"
          exec /bin/bash -c "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/hadoop/hadoop-3-data/grid/log/application_1449046677123_0002/container_1449046677123_0002_01_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog  -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/var/hadoop/hadoop-3-data/grid/log/application_1449046677123_0002/container_1449046677123_0002_01_000001/stdout 2>/var/hadoop/hadoop-3-data/grid/log/application_1449046677123_0002/container_1449046677123_0002_01_000001/stderr "
          
          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new patch to address Sidharta Seethana 's comments. Wangda Tan - Since debug information fetch script (like copy script and list files) is at the end of launch_container.sh, is it possible that a container is killed so such script cannot be executed? It's not at the end - it's just before the actually container process is launched so if we reach a stage where we are ready to call launch_container.sh it should almost always be run. This is what the relevant lines from launch_container.sh look like with the patch: echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>> "/ var /hadoop/hadoop-3-data/grid/log/application_1449046677123_0002/container_1449046677123_0002_01_000001/directory.info" find -L . -maxdepth 5 -type l -ls 1>> "/ var /hadoop/hadoop-3-data/grid/log/application_1449046677123_0002/container_1449046677123_0002_01_000001/directory.info" exec /bin/bash -c "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/ var /hadoop/hadoop-3-data/grid/log/application_1449046677123_0002/container_1449046677123_0002_01_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/ var /hadoop/hadoop-3-data/grid/log/application_1449046677123_0002/container_1449046677123_0002_01_000001/stdout 2>/ var /hadoop/hadoop-3-data/grid/log/application_1449046677123_0002/container_1449046677123_0002_01_000001/stderr "
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 23s trunk passed
          +1 compile 2m 6s trunk passed with JDK v1.8.0_66
          +1 compile 2m 20s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 33s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 3m 49s trunk passed
          +1 javadoc 1m 27s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 49s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 26s the patch passed
          +1 compile 2m 2s the patch passed with JDK v1.8.0_66
          +1 javac 2m 2s the patch passed
          +1 compile 2m 18s the patch passed with JDK v1.7.0_85
          +1 javac 2m 18s the patch passed
          -1 checkstyle 0m 29s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 358, now 359).
          +1 mvnsite 1m 34s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 4m 14s the patch passed
          +1 javadoc 1m 31s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 52s the patch passed with JDK v1.7.0_85
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          -1 unit 9m 0s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          +1 unit 2m 15s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
          +1 unit 9m 20s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          68m 10s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775779/YARN-4309.006.patch
          JIRA Issue YARN-4309
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 52b36b60ac8a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e84d6ca
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9861/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9861/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9861/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9861/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9861/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 23s trunk passed +1 compile 2m 6s trunk passed with JDK v1.8.0_66 +1 compile 2m 20s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 33s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 3m 49s trunk passed +1 javadoc 1m 27s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 49s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 26s the patch passed +1 compile 2m 2s the patch passed with JDK v1.8.0_66 +1 javac 2m 2s the patch passed +1 compile 2m 18s the patch passed with JDK v1.7.0_85 +1 javac 2m 18s the patch passed -1 checkstyle 0m 29s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 358, now 359). +1 mvnsite 1m 34s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 14s the patch passed +1 javadoc 1m 31s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 52s the patch passed with JDK v1.7.0_85 +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. -1 unit 9m 0s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. +1 unit 2m 15s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. +1 unit 9m 20s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 68m 10s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775779/YARN-4309.006.patch JIRA Issue YARN-4309 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 52b36b60ac8a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e84d6ca findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9861/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9861/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9861/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9861/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9861/console This message was automatically generated.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Hi Varun Vasudev, like I mentioned in my previous comment : maybe we should add a note in the (code) comments about symlinks being followed out of the current directory ?

          thanks.

          Show
          sidharta-s Sidharta Seethana added a comment - Hi Varun Vasudev , like I mentioned in my previous comment : maybe we should add a note in the (code) comments about symlinks being followed out of the current directory ? thanks.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new patch with clarifications on following symlinks in the comments and yarn-default.xml .

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new patch with clarifications on following symlinks in the comments and yarn-default.xml .
          Hide
          leftnoteasy Wangda Tan added a comment -

          Varun Vasudev,
          Thanks for reply, make sense to me.

          Few comments:

          • Could you make sure container process will be launched even if copy script or list folder command fails?
          • Could you add echo command (something like echo "Printing container launch debug info...") to container_launch.sh? (After following "if")
            362	    if (getConf() != null && getConf().getBoolean(
            363	        YarnConfiguration.NM_LOG_CONTAINER_DEBUG_INFO,
            364	        YarnConfiguration.DEFAULT_NM_LOG_CONTAINER_DEBUG_INFO)) {
            
          • Add a test to verify log aggregation result contains such debugging output?
          • Could you upload a sample container_launch.sh for easier review?
          Show
          leftnoteasy Wangda Tan added a comment - Varun Vasudev , Thanks for reply, make sense to me. Few comments: Could you make sure container process will be launched even if copy script or list folder command fails? Could you add echo command (something like echo "Printing container launch debug info...") to container_launch.sh? (After following "if") 362 if (getConf() != null && getConf().getBoolean( 363 YarnConfiguration.NM_LOG_CONTAINER_DEBUG_INFO, 364 YarnConfiguration.DEFAULT_NM_LOG_CONTAINER_DEBUG_INFO)) { Add a test to verify log aggregation result contains such debugging output? Could you upload a sample container_launch.sh for easier review?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 34s trunk passed
          +1 compile 1m 52s trunk passed with JDK v1.8.0_66
          +1 compile 2m 9s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 28s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 3m 29s trunk passed
          +1 javadoc 1m 21s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 35s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 21s the patch passed
          +1 compile 1m 44s the patch passed with JDK v1.8.0_66
          +1 javac 1m 44s the patch passed
          +1 compile 2m 3s the patch passed with JDK v1.7.0_91
          +1 javac 2m 3s the patch passed
          -1 checkstyle 0m 26s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 358, now 359).
          +1 mvnsite 1m 29s the patch passed
          +1 mvneclipse 0m 37s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 50s the patch passed
          +1 javadoc 1m 20s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 35s the patch passed with JDK v1.7.0_91
          +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 51s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 32s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 6s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          +1 unit 9m 7s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          64m 28s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776032/YARN-4309.007.patch
          JIRA Issue YARN-4309
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 8217cb3439ac 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 65f3952
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9882/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9882/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 75MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9882/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 34s trunk passed +1 compile 1m 52s trunk passed with JDK v1.8.0_66 +1 compile 2m 9s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 28s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 3m 29s trunk passed +1 javadoc 1m 21s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 35s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 21s the patch passed +1 compile 1m 44s the patch passed with JDK v1.8.0_66 +1 javac 1m 44s the patch passed +1 compile 2m 3s the patch passed with JDK v1.7.0_91 +1 javac 2m 3s the patch passed -1 checkstyle 0m 26s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 358, now 359). +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 50s the patch passed +1 javadoc 1m 20s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 35s the patch passed with JDK v1.7.0_91 +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 1m 51s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 32s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 6s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. +1 unit 9m 7s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 64m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776032/YARN-4309.007.patch JIRA Issue YARN-4309 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 8217cb3439ac 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 65f3952 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9882/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9882/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9882/console This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new patch to address comments by Wangda Tan.

          Could you make sure container process will be launched even if copy script or list folder command fails?

          Fixed.

          Could you add echo command (something like echo "Printing container launch debug info...") to container_launch.sh?

          The echo will end up being captured by the ContainerExecutor and logged in the NM log. Any particular reason you want to print this line?

          Add a test to verify log aggregation result contains such debugging output?

          This would require essentially launching a container and waiting for log aggregation to occur. I'm not sure it will add anything.

          Could you upload a sample container_launch.sh for easier review?

          This is using the yarn logs command with the feature enabled -

          LogType:launch_container.sh
          Log Upload Time:Mon Dec 07 22:43:44 +0530 2015
          LogLength:5042
          Log Contents:
          #!/bin/bash
          
          export JAVA_HOME=${JAVA_HOME:-"/usr/lib/jvm/java-1.7.0-openjdk-amd64"}
          export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
          "
          export NM_HOST="ubuntu"
          export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/var/hadoop/hadoop-3.0.0-SNAPSHOT"}
          export HADOOP_ROOT_LOGGER="INFO,console"
          export JVM_PID="$$"
          export STDERR_LOGFILE_ENV="/var/hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/stderr"
          export PWD="/var/hadoop/hadoop-3-data/grid/local/usercache/varun/appcache/application_1449508378135_0001/container_1449508378135_0001_01_000003"
          export NM_PORT="39813"
          export LOGNAME="varun"
          export MALLOC_ARENA_MAX="4"
          export LD_LIBRARY_PATH="$PWD:/var/hadoop/hadoop-3.0.0-SNAPSHOT/lib/native"
          export LOG_DIRS="/var/hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003,/var/hadoop/hadoop-3-data/grid2/log/application_1449508378135_0001/container_1449508378135_0001_01_000003"
          export NM_HTTP_PORT="8042"
          export SHELL="/bin/bash"
          export LOCAL_DIRS="/var/hadoop/hadoop-3-data/grid/local/usercache/varun/appcache/application_1449508378135_0001"
          export HADOOP_COMMON_HOME=${HADOOP_COMMON_HOME:-"/var/hadoop/hadoop-3.0.0-SNAPSHOT"}
          export HADOOP_TOKEN_FILE_LOCATION="/var/hadoop/hadoop-3-data/grid/local/usercache/varun/appcache/application_1449508378135_0001/container_1449508378135_0001_01_000003/container_tokens"
          export CLASSPATH="$PWD:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:$PWD/*"
          export STDOUT_LOGFILE_ENV="/var/hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/stdout"
          export USER="varun"
          export HADOOP_CLIENT_OPTS="-Xmx512m -Xmx512m  "
          export HADOOP_HDFS_HOME=${HADOOP_HDFS_HOME:-"/var/hadoop/hadoop-3.0.0-SNAPSHOT"}
          export CONTAINER_ID="container_1449508378135_0001_01_000003"
          export HOME="/home/"
          export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/var/hadoop/hadoop-3.0.0-SNAPSHOT/conf"}
          ln -sf "/var/hadoop/hadoop-3-data/grid/local/usercache/varun/appcache/application_1449508378135_0001/filecache/10/job.jar" "job.jar"
          hadoop_shell_errorcode=$?
          if [ $hadoop_shell_errorcode -ne 0 ]
          then
            exit $hadoop_shell_errorcode
          fi
          ln -sf "/var/hadoop/hadoop-3-data/grid/local/usercache/varun/appcache/application_1449508378135_0001/filecache/13/job.xml" "job.xml"
          hadoop_shell_errorcode=$?
          if [ $hadoop_shell_errorcode -ne 0 ]
          then
            exit $hadoop_shell_errorcode
          fi
          cp "launch_container.sh" "/var/hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/launch_container.sh"
          chmod 640 "/var/hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/launch_container.sh"
          echo "ls -l:" 1>"/var/hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/directory.info"
          ls -l 1>>"/var/hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/directory.info"
          echo "find -L . -maxdepth 5 -ls:" 1>>"/var/hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/directory.info"
          find -L . -maxdepth 5 -ls 1>>"/var/hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/directory.info"
          echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/var/hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/directory.info"
          find -L . -maxdepth 5 -type l -ls 1>>"/var/hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/directory.info"
          exec /bin/bash -c "$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN   -Xmx820m -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Dyarn.app.mapreduce.shuffle.logger=INFO,shuffleCLA -Dyarn.app.mapreduce.shuffle.logfile=syslog.shuffle -Dyarn.app.mapreduce.shuffle.log.filesize=0 -Dyarn.app.mapreduce.shuffle.log.backups=0 org.apache.hadoop.mapred.YarnChild 127.0.1.1 36966 attempt_1449508378135_0001_r_000000_0 3 1>/var/hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/stdout 2>/var/hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/stderr "
          hadoop_shell_errorcode=$?
          if [ $hadoop_shell_errorcode -ne 0 ]
          then
            exit $hadoop_shell_errorcode
          fi
          End of LogType:launch_container.sh
          
          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new patch to address comments by Wangda Tan . Could you make sure container process will be launched even if copy script or list folder command fails? Fixed. Could you add echo command (something like echo "Printing container launch debug info...") to container_launch.sh? The echo will end up being captured by the ContainerExecutor and logged in the NM log. Any particular reason you want to print this line? Add a test to verify log aggregation result contains such debugging output? This would require essentially launching a container and waiting for log aggregation to occur. I'm not sure it will add anything. Could you upload a sample container_launch.sh for easier review? This is using the yarn logs command with the feature enabled - LogType:launch_container.sh Log Upload Time:Mon Dec 07 22:43:44 +0530 2015 LogLength:5042 Log Contents: #!/bin/bash export JAVA_HOME=${JAVA_HOME:- "/usr/lib/jvm/java-1.7.0-openjdk-amd64" } export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= " export NM_HOST= "ubuntu" export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:- "/ var /hadoop/hadoop-3.0.0-SNAPSHOT" } export HADOOP_ROOT_LOGGER= "INFO,console" export JVM_PID= "$$" export STDERR_LOGFILE_ENV= "/ var /hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/stderr" export PWD= "/ var /hadoop/hadoop-3-data/grid/local/usercache/varun/appcache/application_1449508378135_0001/container_1449508378135_0001_01_000003" export NM_PORT= "39813" export LOGNAME= "varun" export MALLOC_ARENA_MAX= "4" export LD_LIBRARY_PATH= "$PWD:/ var /hadoop/hadoop-3.0.0-SNAPSHOT/lib/ native " export LOG_DIRS= "/ var /hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003,/ var /hadoop/hadoop-3-data/grid2/log/application_1449508378135_0001/container_1449508378135_0001_01_000003" export NM_HTTP_PORT= "8042" export SHELL= "/bin/bash" export LOCAL_DIRS= "/ var /hadoop/hadoop-3-data/grid/local/usercache/varun/appcache/application_1449508378135_0001" export HADOOP_COMMON_HOME=${HADOOP_COMMON_HOME:- "/ var /hadoop/hadoop-3.0.0-SNAPSHOT" } export HADOOP_TOKEN_FILE_LOCATION= "/ var /hadoop/hadoop-3-data/grid/local/usercache/varun/appcache/application_1449508378135_0001/container_1449508378135_0001_01_000003/container_tokens" export CLASSPATH= "$PWD:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:$PWD/*" export STDOUT_LOGFILE_ENV= "/ var /hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/stdout" export USER= "varun" export HADOOP_CLIENT_OPTS= "-Xmx512m -Xmx512m " export HADOOP_HDFS_HOME=${HADOOP_HDFS_HOME:- "/ var /hadoop/hadoop-3.0.0-SNAPSHOT" } export CONTAINER_ID= "container_1449508378135_0001_01_000003" export HOME= "/home/" export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:- "/ var /hadoop/hadoop-3.0.0-SNAPSHOT/conf" } ln -sf "/ var /hadoop/hadoop-3-data/grid/local/usercache/varun/appcache/application_1449508378135_0001/filecache/10/job.jar" "job.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/ var /hadoop/hadoop-3-data/grid/local/usercache/varun/appcache/application_1449508378135_0001/filecache/13/job.xml" "job.xml" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi cp "launch_container.sh" "/ var /hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/launch_container.sh" chmod 640 "/ var /hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/launch_container.sh" echo "ls -l:" 1> "/ var /hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/directory.info" ls -l 1>> "/ var /hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/directory.info" echo "find -L . -maxdepth 5 -ls:" 1>> "/ var /hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/directory.info" find -L . -maxdepth 5 -ls 1>> "/ var /hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/directory.info" echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>> "/ var /hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/directory.info" find -L . -maxdepth 5 -type l -ls 1>> "/ var /hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/directory.info" exec /bin/bash -c "$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack= true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/ var /hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Dyarn.app.mapreduce.shuffle.logger=INFO,shuffleCLA -Dyarn.app.mapreduce.shuffle.logfile=syslog.shuffle -Dyarn.app.mapreduce.shuffle.log.filesize=0 -Dyarn.app.mapreduce.shuffle.log.backups=0 org.apache.hadoop.mapred.YarnChild 127.0.1.1 36966 attempt_1449508378135_0001_r_000000_0 3 1>/ var /hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/stdout 2>/ var /hadoop/hadoop-3-data/grid/log/application_1449508378135_0001/container_1449508378135_0001_01_000003/stderr " hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi End of LogType:launch_container.sh
          Hide
          aw Allen Wittenauer added a comment - - edited

          (As a sidenote, that launch script has at least two serious bugs in it. I suspect it's only working based upon env var inheritance. I'll file a jira to fix 'em I guess.)

          Show
          aw Allen Wittenauer added a comment - - edited (As a sidenote, that launch script has at least two serious bugs in it. I suspect it's only working based upon env var inheritance. I'll file a jira to fix 'em I guess.)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 2s trunk passed
          +1 compile 2m 4s trunk passed with JDK v1.8.0_66
          +1 compile 2m 17s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 34s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 3m 48s trunk passed
          +1 javadoc 1m 31s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 49s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 31s the patch passed
          +1 compile 2m 15s the patch passed with JDK v1.8.0_66
          +1 javac 2m 15s the patch passed
          +1 compile 2m 16s the patch passed with JDK v1.7.0_85
          +1 javac 2m 16s the patch passed
          -1 checkstyle 0m 29s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 358, now 359).
          +1 mvnsite 1m 34s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 4m 9s the patch passed
          +1 javadoc 1m 27s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 58s the patch passed with JDK v1.7.0_85
          +1 unit 0m 27s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 8s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 9m 18s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 27s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          +1 unit 2m 22s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
          +1 unit 9m 30s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          68m 40s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776134/YARN-4309.008.patch
          JIRA Issue YARN-4309
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux fff8fc3650b5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 01a641b
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9887/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9887/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9887/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 2s trunk passed +1 compile 2m 4s trunk passed with JDK v1.8.0_66 +1 compile 2m 17s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 34s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 3m 48s trunk passed +1 javadoc 1m 31s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 49s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 31s the patch passed +1 compile 2m 15s the patch passed with JDK v1.8.0_66 +1 javac 2m 15s the patch passed +1 compile 2m 16s the patch passed with JDK v1.7.0_85 +1 javac 2m 16s the patch passed -1 checkstyle 0m 29s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 358, now 359). +1 mvnsite 1m 34s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 4m 9s the patch passed +1 javadoc 1m 27s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 58s the patch passed with JDK v1.7.0_85 +1 unit 0m 27s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 8s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 9m 18s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 27s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. +1 unit 2m 22s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. +1 unit 9m 30s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 68m 40s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776134/YARN-4309.008.patch JIRA Issue YARN-4309 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux fff8fc3650b5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 01a641b findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9887/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9887/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9887/console This message was automatically generated.
          Hide
          ivanmi Ivan Mitic added a comment -

          Thanks Varun Vasudev, the proposal and the patch look good overall. I have a few mainly minor comments below, mostly related to Windows side changes.

          1.

              public abstract void copyDebugInformation(Path src, Path dst)
                  throws IOException;
              public abstract void listDebugInformation(Path output) throws IOException;
          

          I think it would be helpful to document what the methods are supposed to do.

          2.

              @Override
              public void copyDebugInformation(Path src, Path dest)
                  throws IOException {
                // no need to worry about permissions - in secure mode
                // WindowsSecureContainerExecutor will set permissions
                // to allow NM to read the file
                lineWithLenCheck(String.format("cp \"%s\" \"%s\"", src.toString(),
                    dest.toString()));
                errorCheck();
              }
          

          Do we want to remove the error check above, to be consistent with Linux, and to avoid failing due to a "logging" failure?
          Also, cp command does not exist on Windows. Please use copy instead.

          3. Why do you have both "dir" and "dir /AL /S" on Windows? Can you please include an inline comment with rationale.

          4. In copyDebugInformation() you are also doing a chmod() internally. Wondering this this command should be injected by the call site given that only the caller has context on what the destination is and whether special permission handling is needed. It might be possible to change the method to only accept src and copy the file to the current folder, in which case it might be fine to use chmod() given that there is an assumption on what the current folder is. Just a thought you make the call.

          Show
          ivanmi Ivan Mitic added a comment - Thanks Varun Vasudev , the proposal and the patch look good overall. I have a few mainly minor comments below, mostly related to Windows side changes. 1. public abstract void copyDebugInformation(Path src, Path dst) throws IOException; public abstract void listDebugInformation(Path output) throws IOException; I think it would be helpful to document what the methods are supposed to do. 2. @Override public void copyDebugInformation(Path src, Path dest) throws IOException { // no need to worry about permissions - in secure mode // WindowsSecureContainerExecutor will set permissions // to allow NM to read the file lineWithLenCheck( String .format( "cp \" %s\ " \" %s\"", src.toString(), dest.toString())); errorCheck(); } Do we want to remove the error check above, to be consistent with Linux, and to avoid failing due to a "logging" failure? Also, cp command does not exist on Windows. Please use copy instead. 3. Why do you have both "dir" and "dir /AL /S" on Windows? Can you please include an inline comment with rationale. 4. In copyDebugInformation() you are also doing a chmod() internally. Wondering this this command should be injected by the call site given that only the caller has context on what the destination is and whether special permission handling is needed. It might be possible to change the method to only accept src and copy the file to the current folder, in which case it might be fine to use chmod() given that there is an assumption on what the current folder is. Just a thought you make the call.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Varun Vasudev.

          The echo will end up being captured by the ContainerExecutor and logged in the NM log. Any particular reason you want to print this line?

          I meant to print comment to the generated container_launch.sh for better readability. Such as:

          (container_launch.sh)
          
          # Print debug information of container launch 
          ......
          
          # Launch container process
          ......
          
          Show
          leftnoteasy Wangda Tan added a comment - Varun Vasudev . The echo will end up being captured by the ContainerExecutor and logged in the NM log. Any particular reason you want to print this line? I meant to print comment to the generated container_launch.sh for better readability. Such as: (container_launch.sh) # Print debug information of container launch ...... # Launch container process ......
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the reviews Ivan Mitic and Wangda Tan.

          I think it would be helpful to document what the methods are supposed to do.

          Fixed.

          Do we want to remove the error check above, to be consistent with Linux, and to avoid failing due to a "logging" failure? Also, cp command does not exist on Windows. Please use copy instead.

          Fixed.

          Why do you have both "dir" and "dir /AL /S" on Windows? Can you please include an inline comment with rationale.

          The original intent was to try to detect broken symlinks but I'm not sure if that's possible using the dir command. I've removed the dir /AL /S command.

          In copyDebugInformation() you are also doing a chmod() internally. Wondering this this command should be injected by the call site given that only the caller has context on what the destination is and whether special permission handling is needed. It might be possible to change the method to only accept src and copy the file to the current folder, in which case it might be fine to use chmod() given that there is an assumption on what the current folder is. Just a thought you make the call.

          Good point. Copying the file to the current folder doesn't work because the container launch script runs in the container work dir and we want these files to be uploaded as part of log aggregation. I've just added a check to make sure the path is absolute before attempting the chmod.

          I meant to print comment to the generated container_launch.sh for better readability. Such as:

          Fixed.

          Show
          vvasudev Varun Vasudev added a comment - Thanks for the reviews Ivan Mitic and Wangda Tan . I think it would be helpful to document what the methods are supposed to do. Fixed. Do we want to remove the error check above, to be consistent with Linux, and to avoid failing due to a "logging" failure? Also, cp command does not exist on Windows. Please use copy instead. Fixed. Why do you have both "dir" and "dir /AL /S" on Windows? Can you please include an inline comment with rationale. The original intent was to try to detect broken symlinks but I'm not sure if that's possible using the dir command. I've removed the dir /AL /S command. In copyDebugInformation() you are also doing a chmod() internally. Wondering this this command should be injected by the call site given that only the caller has context on what the destination is and whether special permission handling is needed. It might be possible to change the method to only accept src and copy the file to the current folder, in which case it might be fine to use chmod() given that there is an assumption on what the current folder is. Just a thought you make the call. Good point. Copying the file to the current folder doesn't work because the container launch script runs in the container work dir and we want these files to be uploaded as part of log aggregation. I've just added a check to make sure the path is absolute before attempting the chmod. I meant to print comment to the generated container_launch.sh for better readability. Such as: Fixed.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 59s trunk passed
          +1 compile 2m 2s trunk passed with JDK v1.8.0_66
          +1 compile 2m 18s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 32s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 3m 55s trunk passed
          +1 javadoc 1m 30s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 49s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 26s the patch passed
          +1 compile 2m 4s the patch passed with JDK v1.8.0_66
          +1 javac 2m 4s the patch passed
          +1 compile 2m 19s the patch passed with JDK v1.7.0_85
          +1 javac 2m 19s the patch passed
          -1 checkstyle 0m 28s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 359, now 359).
          +1 mvnsite 1m 33s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 4m 14s the patch passed
          +1 javadoc 1m 29s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 49s the patch passed with JDK v1.7.0_85
          +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 1s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 48s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          +1 unit 2m 16s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
          +1 unit 9m 17s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85.
          -1 asflicense 0m 20s Patch generated 3 ASF License warnings.
          67m 22s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776345/YARN-4309.009.patch
          JIRA Issue YARN-4309
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux a51fadc230bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d7b3f8d
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9901/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9901/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/9901/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9901/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 59s trunk passed +1 compile 2m 2s trunk passed with JDK v1.8.0_66 +1 compile 2m 18s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 32s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 3m 55s trunk passed +1 javadoc 1m 30s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 49s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 26s the patch passed +1 compile 2m 4s the patch passed with JDK v1.8.0_66 +1 javac 2m 4s the patch passed +1 compile 2m 19s the patch passed with JDK v1.7.0_85 +1 javac 2m 19s the patch passed -1 checkstyle 0m 28s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 359, now 359). +1 mvnsite 1m 33s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 14s the patch passed +1 javadoc 1m 29s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 49s the patch passed with JDK v1.7.0_85 +1 unit 0m 25s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 1s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 48s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. +1 unit 2m 16s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. +1 unit 9m 17s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85. -1 asflicense 0m 20s Patch generated 3 ASF License warnings. 67m 22s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776345/YARN-4309.009.patch JIRA Issue YARN-4309 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux a51fadc230bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d7b3f8d findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9901/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9901/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/9901/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9901/console This message was automatically generated.
          Hide
          ivanmi Ivan Mitic added a comment -

          Thanks Varun Vasudev. Latest patch looks good, I am +1 on the Windows side changes. Please also have someone actively working on Yarn to +1 on the overall approach and Linux side.

          Show
          ivanmi Ivan Mitic added a comment - Thanks Varun Vasudev . Latest patch looks good, I am +1 on the Windows side changes. Please also have someone actively working on Yarn to +1 on the overall approach and Linux side.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks reviews from Ivan Mitic,

          I've tried to deploy Hadoop locally with this patch, it works. The only comment from my side is:

          Do you think is it better to rename yarn.nodemanager.log-container-debug-info to yarn.nodemanager.log-container-debug-info.enabled. Since yarn.nodemanager.log-container-debug-info doesn't show it's a boolean.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks reviews from Ivan Mitic , I've tried to deploy Hadoop locally with this patch, it works. The only comment from my side is: Do you think is it better to rename yarn.nodemanager.log-container-debug-info to yarn.nodemanager.log-container-debug-info.enabled . Since yarn.nodemanager.log-container-debug-info doesn't show it's a boolean.
          Hide
          vvasudev Varun Vasudev added a comment -

          Do you think is it better to rename yarn.nodemanager.log-container-debug-info to yarn.nodemanager.log-container-debug-info.enabled. Since yarn.nodemanager.log-container-debug-info doesn't show it's a boolean.

          Good point. Fixed.

          Show
          vvasudev Varun Vasudev added a comment - Do you think is it better to rename yarn.nodemanager.log-container-debug-info to yarn.nodemanager.log-container-debug-info.enabled. Since yarn.nodemanager.log-container-debug-info doesn't show it's a boolean. Good point. Fixed.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 16s trunk passed
          +1 compile 2m 11s trunk passed with JDK v1.8.0_66
          +1 compile 2m 19s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 37s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 3m 49s trunk passed
          +1 javadoc 1m 29s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 46s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 27s the patch passed
          +1 compile 2m 11s the patch passed with JDK v1.8.0_66
          +1 javac 2m 11s the patch passed
          +1 compile 2m 17s the patch passed with JDK v1.7.0_91
          +1 javac 2m 17s the patch passed
          -1 checkstyle 0m 29s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 358, now 358).
          +1 mvnsite 1m 33s the patch passed
          +1 mvneclipse 0m 40s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 4m 10s the patch passed
          +1 javadoc 1m 27s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 46s the patch passed with JDK v1.7.0_91
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 0s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 54s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 17s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          +1 unit 9m 20s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          67m 53s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776777/YARN-4309.010.patch
          JIRA Issue YARN-4309
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 47ffa7fa5f4f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 132478e
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9924/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9924/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9924/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 16s trunk passed +1 compile 2m 11s trunk passed with JDK v1.8.0_66 +1 compile 2m 19s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 37s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 3m 49s trunk passed +1 javadoc 1m 29s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 46s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 27s the patch passed +1 compile 2m 11s the patch passed with JDK v1.8.0_66 +1 javac 2m 11s the patch passed +1 compile 2m 17s the patch passed with JDK v1.7.0_91 +1 javac 2m 17s the patch passed -1 checkstyle 0m 29s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 358, now 358). +1 mvnsite 1m 33s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 10s the patch passed +1 javadoc 1m 27s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 46s the patch passed with JDK v1.7.0_91 +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 0s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 54s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 17s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. +1 unit 9m 20s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 67m 53s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776777/YARN-4309.010.patch JIRA Issue YARN-4309 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 47ffa7fa5f4f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 132478e findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9924/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9924/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9924/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Varun Vasudev, +1. Will commit in a few days if no opposite opinions.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Varun Vasudev , +1. Will commit in a few days if no opposite opinions.
          Hide
          leftnoteasy Wangda Tan added a comment - - edited

          Committed to trunk/branch-2. Thanks Varun Vasudev and review from Steve Loughran/Sidharta Seethana/Allen Wittenauer/Jason Lowe/Karthik Kambatla!

          Show
          leftnoteasy Wangda Tan added a comment - - edited Committed to trunk/branch-2. Thanks Varun Vasudev and review from Steve Loughran / Sidharta Seethana / Allen Wittenauer / Jason Lowe / Karthik Kambatla !
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8962 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8962/)
          YARN-4309. Add container launch related debug information to container (wangda: rev dfcbbddb0963c89c0455d41223427165b9f9e537)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8962 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8962/ ) YARN-4309 . Add container launch related debug information to container (wangda: rev dfcbbddb0963c89c0455d41223427165b9f9e537) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #692 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/692/)
          YARN-4309. Add container launch related debug information to container (wangda: rev dfcbbddb0963c89c0455d41223427165b9f9e537)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #692 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/692/ ) YARN-4309 . Add container launch related debug information to container (wangda: rev dfcbbddb0963c89c0455d41223427165b9f9e537) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Hide
          leftnoteasy Wangda Tan added a comment -

          Committed to branch-2.8.

          Show
          leftnoteasy Wangda Tan added a comment - Committed to branch-2.8.

            People

            • Assignee:
              vvasudev Varun Vasudev
              Reporter:
              vvasudev Varun Vasudev
            • Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development