Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5641

Localizer leaves behind tarballs after container is complete

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The localizer sometimes fails to clean up extracted tarballs leaving large footprints that persist on the nodes indefinitely.

      1. YARN-5641.001.patch
        4 kB
        Eric Badger
      2. YARN-5641.002.patch
        5 kB
        Eric Badger
      3. YARN-5641.003.patch
        6 kB
        Eric Badger
      4. YARN-5641.004.patch
        1 kB
        Eric Badger
      5. YARN-5641.005.patch
        3 kB
        Eric Badger
      6. YARN-5641.006.patch
        22 kB
        Eric Badger
      7. YARN-5641.007.patch
        30 kB
        Eric Badger
      8. YARN-5641.008.patch
        27 kB
        Eric Badger
      9. YARN-5641.009.patch
        27 kB
        Eric Badger
      10. YARN-5641.009.patch
        27 kB
        Eric Badger
      11. YARN-5641.010.patch
        27 kB
        Eric Badger
      12. YARN-5641.010-b2.patch
        27 kB
        Eric Badger
      13. YARN-5641-branch-2.011.patch
        27 kB
        Eric Badger

        Issue Links

          Activity

          Hide
          ebadger Eric Badger added a comment -

          Thanks, Jason Lowe!

          Show
          ebadger Eric Badger added a comment - Thanks, Jason Lowe !
          Hide
          jlowe Jason Lowe added a comment -

          Thanks, Eric! I committed this to branch-2.

          Show
          jlowe Jason Lowe added a comment - Thanks, Eric! I committed this to branch-2.
          Hide
          jlowe Jason Lowe added a comment -

          +1 for the latest branch-2 patch. I'll fixup the extra import during the commit.

          Show
          jlowe Jason Lowe added a comment - +1 for the latest branch-2 patch. I'll fixup the extra import during the commit.
          Hide
          ebadger Eric Badger added a comment -

          The test failures are unrelated. However, since I got rid of the waitFor() parameters (not supported in Java 7), there is an unused TimeUnit import. The warnings are from a piece of code that wasn't touched.

          Show
          ebadger Eric Badger added a comment - The test failures are unrelated. However, since I got rid of the waitFor() parameters (not supported in Java 7), there is an unused TimeUnit import. The warnings are from a piece of code that wasn't touched.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 14m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 9s Maven dependency ordering for branch
          +1 mvninstall 7m 43s branch-2 passed
          +1 compile 9m 13s branch-2 passed with JDK v1.8.0_121
          +1 compile 8m 20s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 1m 45s branch-2 passed
          +1 mvnsite 1m 52s branch-2 passed
          +1 mvneclipse 0m 34s branch-2 passed
          +1 findbugs 3m 26s branch-2 passed
          +1 javadoc 1m 11s branch-2 passed with JDK v1.8.0_121
          +1 javadoc 1m 24s branch-2 passed with JDK v1.7.0_121
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 7m 12s the patch passed with JDK v1.8.0_121
          -1 javac 7m 12s root-jdk1.8.0_121 with JDK v1.8.0_121 generated 1 new + 879 unchanged - 1 fixed = 880 total (was 880)
          +1 compile 7m 57s the patch passed with JDK v1.7.0_121
          -1 javac 7m 57s root-jdk1.7.0_121 with JDK v1.7.0_121 generated 1 new + 974 unchanged - 1 fixed = 975 total (was 975)
          -0 checkstyle 1m 36s root: The patch generated 6 new + 101 unchanged - 0 fixed = 107 total (was 101)
          +1 mvnsite 1m 39s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 2s the patch passed
          +1 javadoc 1m 28s the patch passed with JDK v1.8.0_121
          +1 javadoc 1m 39s the patch passed with JDK v1.7.0_121
          +1 unit 14m 5s hadoop-common in the patch passed with JDK v1.7.0_121.
          +1 unit 14m 18s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_121.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          156m 11s



          Reason Tests
          JDK v1.8.0_121 Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.fs.TestTrash



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue YARN-5641
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849552/YARN-5641-branch-2.011.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7e0d028ba2d6 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 4a48f0e
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/14761/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_121.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/14761/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_121.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14761/artifact/patchprocess/diff-checkstyle-root.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14761/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14761/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 14m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 9s Maven dependency ordering for branch +1 mvninstall 7m 43s branch-2 passed +1 compile 9m 13s branch-2 passed with JDK v1.8.0_121 +1 compile 8m 20s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 1m 45s branch-2 passed +1 mvnsite 1m 52s branch-2 passed +1 mvneclipse 0m 34s branch-2 passed +1 findbugs 3m 26s branch-2 passed +1 javadoc 1m 11s branch-2 passed with JDK v1.8.0_121 +1 javadoc 1m 24s branch-2 passed with JDK v1.7.0_121 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 7m 12s the patch passed with JDK v1.8.0_121 -1 javac 7m 12s root-jdk1.8.0_121 with JDK v1.8.0_121 generated 1 new + 879 unchanged - 1 fixed = 880 total (was 880) +1 compile 7m 57s the patch passed with JDK v1.7.0_121 -1 javac 7m 57s root-jdk1.7.0_121 with JDK v1.7.0_121 generated 1 new + 974 unchanged - 1 fixed = 975 total (was 975) -0 checkstyle 1m 36s root: The patch generated 6 new + 101 unchanged - 0 fixed = 107 total (was 101) +1 mvnsite 1m 39s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 2s the patch passed +1 javadoc 1m 28s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 39s the patch passed with JDK v1.7.0_121 +1 unit 14m 5s hadoop-common in the patch passed with JDK v1.7.0_121. +1 unit 14m 18s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_121. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 156m 11s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.fs.TestTrash Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue YARN-5641 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849552/YARN-5641-branch-2.011.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7e0d028ba2d6 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 4a48f0e Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/14761/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_121.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/14761/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_121.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14761/artifact/patchprocess/diff-checkstyle-root.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14761/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14761/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          Reopening to run Jenkins on the branch-2 patch.

          Show
          jlowe Jason Lowe added a comment - Reopening to run Jenkins on the branch-2 patch.
          Hide
          ebadger Eric Badger added a comment -

          Changed isAlive() method to private

          Show
          ebadger Eric Badger added a comment - Changed isAlive() method to private
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the prompt response.
          I think the failure is understandable since the backport was clean. Thanks for working on the branch-2 patch.

          Show
          xiaochen Xiao Chen added a comment - Thanks for the prompt response. I think the failure is understandable since the backport was clean. Thanks for working on the branch-2 patch.
          Hide
          jlowe Jason Lowe added a comment -

          The branch-2 patch looks reasonable to me. Nit: the new isAlive method should be private. When fixing this, please post the patch according to the recommended patch naming conventions so we can reopen the JIRA and get the QA bot to run the standard tests on it.

          Show
          jlowe Jason Lowe added a comment - The branch-2 patch looks reasonable to me. Nit: the new isAlive method should be private. When fixing this, please post the patch according to the recommended patch naming conventions so we can reopen the JIRA and get the QA bot to run the standard tests on it.
          Hide
          ebadger Eric Badger added a comment -

          Apologies for the bad Java 7 patch! I'll be sure to compile against Java 7 for anything that's targeted to anything other than trunk from now on. Here's a new patch for branch-2 that fixes the issues.

          Show
          ebadger Eric Badger added a comment - Apologies for the bad Java 7 patch! I'll be sure to compile against Java 7 for anything that's targeted to anything other than trunk from now on. Here's a new patch for branch-2 that fixes the issues.
          Hide
          jlowe Jason Lowe added a comment -

          My apologies, I missed the isAlive method calls on Process which are only available in JDK8. This has been reverted from branch-2. Eric Badger please provide a separate patch for branch-2 that works with JDK7.

          Show
          jlowe Jason Lowe added a comment - My apologies, I missed the isAlive method calls on Process which are only available in JDK8. This has been reverted from branch-2. Eric Badger please provide a separate patch for branch-2 that works with JDK7.
          Hide
          xiaochen Xiao Chen added a comment -

          It seems this commit broke branch-2 compilation. Could you take a look? Thanks.

          https://builds.apache.org/job/PreCommit-HADOOP-Build/11511/artifact/patchprocess/branch-compile-root-jdk1.7.0_121.txt

          Show
          xiaochen Xiao Chen added a comment - It seems this commit broke branch-2 compilation. Could you take a look? Thanks. https://builds.apache.org/job/PreCommit-HADOOP-Build/11511/artifact/patchprocess/branch-compile-root-jdk1.7.0_121.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11174 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11174/)
          YARN-5641. Localizer leaves behind tarballs after container is complete. (jlowe: rev 9e19f758c1950cbcfcd1969461a8a910efca0767)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShell.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11174 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11174/ ) YARN-5641 . Localizer leaves behind tarballs after container is complete. (jlowe: rev 9e19f758c1950cbcfcd1969461a8a910efca0767) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShell.java
          Hide
          jlowe Jason Lowe added a comment -

          Thanks, Eric! I committed this to trunk and branch-2.

          Show
          jlowe Jason Lowe added a comment - Thanks, Eric! I committed this to trunk and branch-2.
          Hide
          jlowe Jason Lowe added a comment -

          +1 for the latest patch. The unit test failures are unrelated. Committing this.

          Show
          jlowe Jason Lowe added a comment - +1 for the latest patch. The unit test failures are unrelated. Committing this.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 30s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 27s Maven dependency ordering for branch
          +1 mvninstall 12m 47s trunk passed
          +1 compile 20m 44s trunk passed
          +1 checkstyle 3m 31s trunk passed
          +1 mvnsite 3m 19s trunk passed
          +1 mvneclipse 0m 36s trunk passed
          +1 findbugs 2m 22s trunk passed
          +1 javadoc 1m 14s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 6s the patch passed
          +1 compile 11m 48s the patch passed
          +1 javac 11m 48s the patch passed
          -0 checkstyle 1m 41s root: The patch generated 5 new + 103 unchanged - 0 fixed = 108 total (was 103)
          +1 mvnsite 1m 48s the patch passed
          +1 mvneclipse 0m 40s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 42s the patch passed
          +1 javadoc 1m 27s the patch passed
          -1 unit 8m 13s hadoop-common in the patch failed.
          +1 unit 13m 30s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 39s The patch does not generate ASF License warnings.
          113m 30s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5641
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849153/YARN-5641.010.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c8371f6d1622 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5a56520
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14752/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14752/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14752/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14752/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 30s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 27s Maven dependency ordering for branch +1 mvninstall 12m 47s trunk passed +1 compile 20m 44s trunk passed +1 checkstyle 3m 31s trunk passed +1 mvnsite 3m 19s trunk passed +1 mvneclipse 0m 36s trunk passed +1 findbugs 2m 22s trunk passed +1 javadoc 1m 14s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 6s the patch passed +1 compile 11m 48s the patch passed +1 javac 11m 48s the patch passed -0 checkstyle 1m 41s root: The patch generated 5 new + 103 unchanged - 0 fixed = 108 total (was 103) +1 mvnsite 1m 48s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 42s the patch passed +1 javadoc 1m 27s the patch passed -1 unit 8m 13s hadoop-common in the patch failed. +1 unit 13m 30s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 113m 30s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5641 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849153/YARN-5641.010.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c8371f6d1622 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5a56520 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14752/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14752/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14752/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14752/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 2m 7s Maven dependency ordering for branch
          +1 mvninstall 13m 35s trunk passed
          +1 compile 12m 48s trunk passed
          +1 checkstyle 1m 34s trunk passed
          +1 mvnsite 1m 33s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 2m 16s trunk passed
          +1 javadoc 1m 15s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 4s the patch passed
          +1 compile 11m 25s the patch passed
          +1 javac 11m 25s the patch passed
          -0 checkstyle 3m 17s root: The patch generated 5 new + 103 unchanged - 0 fixed = 108 total (was 103)
          +1 mvnsite 4m 28s the patch passed
          +1 mvneclipse 0m 44s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 40s the patch passed
          +1 javadoc 1m 24s the patch passed
          -1 unit 12m 1s hadoop-common in the patch failed.
          -1 unit 17m 41s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 43s The patch does not generate ASF License warnings.
          116m 21s



          Reason Tests
          Failed junit tests hadoop.fs.TestTrash
            hadoop.yarn.server.nodemanager.TestNodeManagerResync



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5641
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849153/YARN-5641.010.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8b132c61d0ca 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a33ce45
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14744/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14744/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14744/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14744/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14744/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 2m 7s Maven dependency ordering for branch +1 mvninstall 13m 35s trunk passed +1 compile 12m 48s trunk passed +1 checkstyle 1m 34s trunk passed +1 mvnsite 1m 33s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 2m 16s trunk passed +1 javadoc 1m 15s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 4s the patch passed +1 compile 11m 25s the patch passed +1 javac 11m 25s the patch passed -0 checkstyle 3m 17s root: The patch generated 5 new + 103 unchanged - 0 fixed = 108 total (was 103) +1 mvnsite 4m 28s the patch passed +1 mvneclipse 0m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 40s the patch passed +1 javadoc 1m 24s the patch passed -1 unit 12m 1s hadoop-common in the patch failed. -1 unit 17m 41s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 43s The patch does not generate ASF License warnings. 116m 21s Reason Tests Failed junit tests hadoop.fs.TestTrash   hadoop.yarn.server.nodemanager.TestNodeManagerResync Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5641 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849153/YARN-5641.010.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8b132c61d0ca 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a33ce45 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14744/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14744/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14744/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14744/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14744/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Somehow change didn't get into last patch. Double checked that it's in this one.

          Show
          ebadger Eric Badger added a comment - Somehow change didn't get into last patch. Double checked that it's in this one.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 12m 43s trunk passed
          +1 compile 12m 17s trunk passed
          +1 checkstyle 1m 34s trunk passed
          +1 mvnsite 1m 31s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 2m 14s trunk passed
          +1 javadoc 1m 14s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 4s the patch passed
          +1 compile 10m 41s the patch passed
          +1 javac 10m 41s the patch passed
          -0 checkstyle 1m 37s root: The patch generated 6 new + 103 unchanged - 0 fixed = 109 total (was 103)
          +1 mvnsite 1m 36s the patch passed
          +1 mvneclipse 0m 45s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 36s the patch passed
          +1 javadoc 1m 22s the patch passed
          +1 unit 7m 50s hadoop-common in the patch passed.
          +1 unit 13m 18s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 38s The patch does not generate ASF License warnings.
          98m 59s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5641
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849134/YARN-5641.009.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a9b9a34db3c5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a33ce45
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14742/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14742/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14742/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 12m 43s trunk passed +1 compile 12m 17s trunk passed +1 checkstyle 1m 34s trunk passed +1 mvnsite 1m 31s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 2m 14s trunk passed +1 javadoc 1m 14s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 4s the patch passed +1 compile 10m 41s the patch passed +1 javac 10m 41s the patch passed -0 checkstyle 1m 37s root: The patch generated 6 new + 103 unchanged - 0 fixed = 109 total (was 103) +1 mvnsite 1m 36s the patch passed +1 mvneclipse 0m 45s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 36s the patch passed +1 javadoc 1m 22s the patch passed +1 unit 7m 50s hadoop-common in the patch passed. +1 unit 13m 18s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 98m 59s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5641 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849134/YARN-5641.009.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a9b9a34db3c5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a33ce45 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14742/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14742/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14742/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Removed unnecessary import and fixed checkstyle

          Show
          ebadger Eric Badger added a comment - Removed unnecessary import and fixed checkstyle
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for updating the patch! There's another unnecessary dependency triggered by using SynchronizedSet from org.apache.commons.collections instead of the Java built-in available in java.util.Collections. It would be preferable to use the built-in so we don't add another unnecessary dependency reference.

          Show
          jlowe Jason Lowe added a comment - Thanks for updating the patch! There's another unnecessary dependency triggered by using SynchronizedSet from org.apache.commons.collections instead of the Java built-in available in java.util.Collections. It would be preferable to use the built-in so we don't add another unnecessary dependency reference.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 12s Maven dependency ordering for branch
          +1 mvninstall 12m 35s trunk passed
          +1 compile 9m 35s trunk passed
          +1 checkstyle 1m 35s trunk passed
          +1 mvnsite 1m 40s trunk passed
          +1 mvneclipse 0m 36s trunk passed
          +1 findbugs 2m 22s trunk passed
          +1 javadoc 1m 18s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 1m 18s the patch passed
          +1 compile 10m 43s the patch passed
          +1 javac 10m 43s the patch passed
          -0 checkstyle 1m 37s root: The patch generated 6 new + 103 unchanged - 0 fixed = 109 total (was 103)
          +1 mvnsite 1m 46s the patch passed
          +1 mvneclipse 0m 42s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 42s the patch passed
          +1 javadoc 1m 22s the patch passed
          +1 unit 8m 40s hadoop-common in the patch passed.
          +1 unit 13m 16s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 39s The patch does not generate ASF License warnings.
          97m 14s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5641
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848590/YARN-5641.009.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6c10a83a4777 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e015b56
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14722/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14722/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14722/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 12m 35s trunk passed +1 compile 9m 35s trunk passed +1 checkstyle 1m 35s trunk passed +1 mvnsite 1m 40s trunk passed +1 mvneclipse 0m 36s trunk passed +1 findbugs 2m 22s trunk passed +1 javadoc 1m 18s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 18s the patch passed +1 compile 10m 43s the patch passed +1 javac 10m 43s the patch passed -0 checkstyle 1m 37s root: The patch generated 6 new + 103 unchanged - 0 fixed = 109 total (was 103) +1 mvnsite 1m 46s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 42s the patch passed +1 javadoc 1m 22s the patch passed +1 unit 8m 40s hadoop-common in the patch passed. +1 unit 13m 16s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 97m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5641 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848590/YARN-5641.009.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6c10a83a4777 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e015b56 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14722/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14722/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14722/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          It should be either java.util.concurrent.ConcurrentHashMap (and store something bogus for the value) or just wrap it in a SynchronizedSet.

          Wrapped the HashSet inside of a SynchronizedSet.

          I'm curious why the random and localDirs fields of ContainerLocalizerWrapper are marked protected but the others are not? Do they need to be marked protected?

          This was a relic of when I was moving things around trying to limit scope. They don't need to be protected.

          Checkstyle also wants me to make all of the variables private and give them accessor methods instead of doing the wrapper.* access. I can fix this if you want.

          Attaching new patch

          Show
          ebadger Eric Badger added a comment - It should be either java.util.concurrent.ConcurrentHashMap (and store something bogus for the value) or just wrap it in a SynchronizedSet. Wrapped the HashSet inside of a SynchronizedSet. I'm curious why the random and localDirs fields of ContainerLocalizerWrapper are marked protected but the others are not? Do they need to be marked protected? This was a relic of when I was moving things around trying to limit scope. They don't need to be protected. Checkstyle also wants me to make all of the variables private and give them accessor methods instead of doing the wrapper.* access. I can fix this if you want. Attaching new patch
          Hide
          jlowe Jason Lowe added a comment -

          We should not be using org.eclipse.jetty.util.ConcurrentHashSet. That's a dependency we do not want here. It should be either java.util.concurrent.ConcurrentHashMap (and store something bogus for the value) or just wrap it in a SynchronizedSet. The latter is probably preferable since I think concurrent is overkill here. It's not super performance sensitive.

          I'm curious why the random and localDirs fields of ContainerLocalizerWrapper are marked protected but the others are not? Do they need to be marked protected?

          Show
          jlowe Jason Lowe added a comment - We should not be using org.eclipse.jetty.util.ConcurrentHashSet. That's a dependency we do not want here. It should be either java.util.concurrent.ConcurrentHashMap (and store something bogus for the value) or just wrap it in a SynchronizedSet. The latter is probably preferable since I think concurrent is overkill here. It's not super performance sensitive. I'm curious why the random and localDirs fields of ContainerLocalizerWrapper are marked protected but the others are not? Do they need to be marked protected?
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 19s Maven dependency ordering for branch
          +1 mvninstall 14m 29s trunk passed
          +1 compile 13m 4s trunk passed
          +1 checkstyle 1m 41s trunk passed
          +1 mvnsite 1m 44s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 2m 42s trunk passed
          +1 javadoc 1m 21s trunk passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 1m 21s the patch passed
          +1 compile 11m 31s the patch passed
          +1 javac 11m 31s the patch passed
          -0 checkstyle 1m 40s root: The patch generated 7 new + 103 unchanged - 0 fixed = 110 total (was 103)
          +1 mvnsite 1m 45s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 3s the patch passed
          +1 javadoc 1m 22s the patch passed
          +1 unit 8m 31s hadoop-common in the patch passed.
          +1 unit 13m 19s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          104m 16s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5641
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848426/YARN-5641.008.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2d8f4bf5170e 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5d8b80e
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14716/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14716/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14716/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 14m 29s trunk passed +1 compile 13m 4s trunk passed +1 checkstyle 1m 41s trunk passed +1 mvnsite 1m 44s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 2m 42s trunk passed +1 javadoc 1m 21s trunk passed 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 1m 21s the patch passed +1 compile 11m 31s the patch passed +1 javac 11m 31s the patch passed -0 checkstyle 1m 40s root: The patch generated 7 new + 103 unchanged - 0 fixed = 110 total (was 103) +1 mvnsite 1m 45s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 3s the patch passed +1 javadoc 1m 22s the patch passed +1 unit 8m 31s hadoop-common in the patch passed. +1 unit 13m 19s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 104m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5641 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848426/YARN-5641.008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2d8f4bf5170e 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5d8b80e Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14716/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14716/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14716/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Thanks for the review, Jason Lowe! I agree with all of your comments and nits and fixed them in this new patch. I also attempted to fix the checkstyle errors that showed up in the previous Jenkins run. Hopefully I caught all of them.

          Show
          ebadger Eric Badger added a comment - Thanks for the review, Jason Lowe ! I agree with all of your comments and nits and fixed them in this new patch. I also attempted to fix the checkstyle errors that showed up in the previous Jenkins run. Hopefully I caught all of them.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for updating the patch!

          I think getThread and the description is a little vague. It would be more clear if the method were called something like getWaitingThread to indicate this is the thread waiting for the shell command to return, and the Javadoc should be more clear that it is the thread in the runCommand method or null if no thread is waiting for a shell command to complete.

          runCommand needs to set the waiting thread field to null after the subprocess completes.

          removeShell does not seem to be appropriate to be public or even necessary. Shell instances should automatically remove themselves when the subprocess exits, and if the subprocess hasn't exited then they shouldn't be removed otherwise. I don't think it's appropriate for the ContainerLocalizer to be removing shells, or am I missing something?

          FSDownloadWrapper should be package-private.

          HashSet is not thread safe, and multiple threads can be accessing it at the same time (e.g.: one thread in the executor pool may be trying to add itself to the set just as another is trying to call the set's contains method). This needs to either be a synchronized set or a concurrent set.

          Why doesn't the ContainerLocalizerWrapper constructor initialize the spylfs member? It's odd to call the constructor then always have to poke fields in it.

          To help reduce the changes in the patch, I'd recommend creating spylfs, random, and nmProxy local variables to eliminate all the changes that are simply adding the 'wrapper.' prefix.

          Nit: Please use a much smaller check interval on the waitFor calls in the unit test.

          Style Nit: Typically there is no space between the generic class name and the type specifier (e.g.: Map <> should be Map<>)
          Style Nit: Please do not use wildcard imports

          Show
          jlowe Jason Lowe added a comment - Thanks for updating the patch! I think getThread and the description is a little vague. It would be more clear if the method were called something like getWaitingThread to indicate this is the thread waiting for the shell command to return, and the Javadoc should be more clear that it is the thread in the runCommand method or null if no thread is waiting for a shell command to complete. runCommand needs to set the waiting thread field to null after the subprocess completes. removeShell does not seem to be appropriate to be public or even necessary. Shell instances should automatically remove themselves when the subprocess exits, and if the subprocess hasn't exited then they shouldn't be removed otherwise. I don't think it's appropriate for the ContainerLocalizer to be removing shells, or am I missing something? FSDownloadWrapper should be package-private. HashSet is not thread safe, and multiple threads can be accessing it at the same time (e.g.: one thread in the executor pool may be trying to add itself to the set just as another is trying to call the set's contains method). This needs to either be a synchronized set or a concurrent set. Why doesn't the ContainerLocalizerWrapper constructor initialize the spylfs member? It's odd to call the constructor then always have to poke fields in it. To help reduce the changes in the patch, I'd recommend creating spylfs , random , and nmProxy local variables to eliminate all the changes that are simply adding the 'wrapper.' prefix. Nit: Please use a much smaller check interval on the waitFor calls in the unit test. Style Nit: Typically there is no space between the generic class name and the type specifier (e.g.: Map <> should be Map<> ) Style Nit: Please do not use wildcard imports
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 59s Maven dependency ordering for branch
          +1 mvninstall 13m 20s trunk passed
          +1 compile 10m 8s trunk passed
          +1 checkstyle 1m 34s trunk passed
          +1 mvnsite 1m 34s trunk passed
          +1 mvneclipse 0m 35s trunk passed
          +1 findbugs 2m 20s trunk passed
          +1 javadoc 1m 16s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 5s the patch passed
          +1 compile 9m 40s the patch passed
          +1 javac 9m 40s the patch passed
          -0 checkstyle 1m 38s root: The patch generated 43 new + 103 unchanged - 0 fixed = 146 total (was 103)
          +1 mvnsite 1m 37s the patch passed
          +1 mvneclipse 0m 42s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 43s the patch passed
          +1 javadoc 1m 22s the patch passed
          +1 unit 7m 41s hadoop-common in the patch passed.
          +1 unit 13m 8s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          97m 20s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5641
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847872/YARN-5641.007.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b720c1acb45c 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8a0fa0f
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14691/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14691/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14691/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 59s Maven dependency ordering for branch +1 mvninstall 13m 20s trunk passed +1 compile 10m 8s trunk passed +1 checkstyle 1m 34s trunk passed +1 mvnsite 1m 34s trunk passed +1 mvneclipse 0m 35s trunk passed +1 findbugs 2m 20s trunk passed +1 javadoc 1m 16s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 5s the patch passed +1 compile 9m 40s the patch passed +1 javac 9m 40s the patch passed -0 checkstyle 1m 38s root: The patch generated 43 new + 103 unchanged - 0 fixed = 146 total (was 103) +1 mvnsite 1m 37s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 43s the patch passed +1 javadoc 1m 22s the patch passed +1 unit 7m 41s hadoop-common in the patch passed. +1 unit 13m 8s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 97m 20s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5641 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847872/YARN-5641.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b720c1acb45c 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8a0fa0f Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14691/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14691/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14691/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Previous patch did some things that JUnit didn't like. This patch fixes those issues. However, it requires changes to existing tests.

          Show
          ebadger Eric Badger added a comment - Previous patch did some things that JUnit didn't like. This patch fixes those issues. However, it requires changes to existing tests.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 2m 10s Maven dependency ordering for branch
          +1 mvninstall 22m 7s trunk passed
          +1 compile 18m 15s trunk passed
          +1 checkstyle 1m 40s trunk passed
          +1 mvnsite 2m 6s trunk passed
          +1 mvneclipse 0m 44s trunk passed
          +1 findbugs 3m 3s trunk passed
          +1 javadoc 1m 29s trunk passed
          0 mvndep 0m 21s Maven dependency ordering for patch
          +1 mvninstall 1m 31s the patch passed
          +1 compile 12m 27s the patch passed
          +1 javac 12m 27s the patch passed
          -0 checkstyle 1m 46s root: The patch generated 33 new + 103 unchanged - 0 fixed = 136 total (was 103)
          +1 mvnsite 1m 56s the patch passed
          +1 mvneclipse 0m 51s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 46s the patch passed
          +1 javadoc 1m 40s the patch passed
          -1 unit 9m 48s hadoop-common in the patch failed.
          -1 unit 14m 12s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 47s The patch does not generate ASF License warnings.
          125m 21s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.yarn.server.nodemanager.containermanager.localizer.TestContainerLocalizer$FakeTestContainerLocalizer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5641
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847419/YARN-5641.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1ec4223db446 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d3170f9
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14653/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14653/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14653/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14653/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14653/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 2m 10s Maven dependency ordering for branch +1 mvninstall 22m 7s trunk passed +1 compile 18m 15s trunk passed +1 checkstyle 1m 40s trunk passed +1 mvnsite 2m 6s trunk passed +1 mvneclipse 0m 44s trunk passed +1 findbugs 3m 3s trunk passed +1 javadoc 1m 29s trunk passed 0 mvndep 0m 21s Maven dependency ordering for patch +1 mvninstall 1m 31s the patch passed +1 compile 12m 27s the patch passed +1 javac 12m 27s the patch passed -0 checkstyle 1m 46s root: The patch generated 33 new + 103 unchanged - 0 fixed = 136 total (was 103) +1 mvnsite 1m 56s the patch passed +1 mvneclipse 0m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 46s the patch passed +1 javadoc 1m 40s the patch passed -1 unit 9m 48s hadoop-common in the patch failed. -1 unit 14m 12s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 47s The patch does not generate ASF License warnings. 125m 21s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.yarn.server.nodemanager.containermanager.localizer.TestContainerLocalizer$FakeTestContainerLocalizer Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5641 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847419/YARN-5641.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1ec4223db446 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d3170f9 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14653/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14653/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14653/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14653/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14653/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Putting up a new patch with a different direction. The previous patch used a single map to keep track of all subprocesses that were spawned via shell and then killed all of them upon the localizer being shutdown. This was fine under the assumption that there is only a single shell process being run. However, in the case of the tests that failed or multiple localizers running on a single NM, there could be multiple shell subprocesses being run via the same parent process. Having one process blindly kill all other running subprocesses would allow for one localizer to kill the subprocesses of another localizer.

          So, instead, this new patch will have the localizer keep track of which subprocesses were spawned by it and only act on those. The new patch still uses a single static Map in Shell (CHILD_SHELLS) to keep track of all shells, but adds in the support to keep track of their associated parent thread. The idea is that the Shell map will simply return all subprocesses to the caller (localizer) instead of actively killing them. This gives the control over what to do back to the caller. The threads from the localizer threadpool that started the subprocesses log themselves in a Set (localizingThreads) that is a member variable of each ContainerLocalizer. Then, once the localizer is shutting down, it can match the parent threads of all of the shells from the Shell Map (CHILD_SHELLS) against the Set of threads for the current localizer (localizingThreads). The localizer gets to choose what it wants to do with the matching shells and in this case it will kill them.

          There is also a new unit test added that starts up 2 localizers and kills one of them. It checks to make sure that the subprocess of the first localizer was killed, while the subprocess of the second localizer is still running.

          Jason Lowe, could you review this?

          Show
          ebadger Eric Badger added a comment - Putting up a new patch with a different direction. The previous patch used a single map to keep track of all subprocesses that were spawned via shell and then killed all of them upon the localizer being shutdown. This was fine under the assumption that there is only a single shell process being run. However, in the case of the tests that failed or multiple localizers running on a single NM, there could be multiple shell subprocesses being run via the same parent process. Having one process blindly kill all other running subprocesses would allow for one localizer to kill the subprocesses of another localizer. So, instead, this new patch will have the localizer keep track of which subprocesses were spawned by it and only act on those. The new patch still uses a single static Map in Shell ( CHILD_SHELLS ) to keep track of all shells, but adds in the support to keep track of their associated parent thread. The idea is that the Shell map will simply return all subprocesses to the caller (localizer) instead of actively killing them. This gives the control over what to do back to the caller. The threads from the localizer threadpool that started the subprocesses log themselves in a Set ( localizingThreads ) that is a member variable of each ContainerLocalizer . Then, once the localizer is shutting down, it can match the parent threads of all of the shells from the Shell Map ( CHILD_SHELLS ) against the Set of threads for the current localizer ( localizingThreads ). The localizer gets to choose what it wants to do with the matching shells and in this case it will kill them. There is also a new unit test added that starts up 2 localizers and kills one of them. It checks to make sure that the subprocess of the first localizer was killed, while the subprocess of the second localizer is still running. Jason Lowe , could you review this?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 14m 23s trunk passed
          +1 compile 0m 30s trunk passed
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 0m 27s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 49s trunk passed
          +1 javadoc 0m 17s trunk passed
          +1 mvninstall 0m 27s the patch passed
          +1 compile 0m 26s the patch passed
          +1 javac 0m 26s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 24s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 58s the patch passed
          +1 javadoc 0m 17s the patch passed
          -1 unit 14m 39s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          36m 24s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
            hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
            hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5641
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843617/YARN-5641.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1a6d3b6de260 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f121645
          Default Java 1.8.0_111
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14349/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14349/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14349/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 23s trunk passed +1 compile 0m 30s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 49s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 27s the patch passed +1 compile 0m 26s the patch passed +1 javac 0m 26s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 58s the patch passed +1 javadoc 0m 17s the patch passed -1 unit 14m 39s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 36m 24s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManager   hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor   hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5641 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843617/YARN-5641.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1a6d3b6de260 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f121645 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/14349/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14349/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14349/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Jason Lowe, since HADOOP-13709 was committed, we can add in the localizer-specific piece. Here's a patch that attempts to shutdown the thread pool threads, kills all subprocesses, and then waits for the thread pool threads to terminate. If they do not terminate in 10 seconds, then we will call System.exit() to force the shutdown hook to be invoked, which will call destroyAllProcesses() again to make sure that all of the spawned subprocesses have been killed.

          Show
          ebadger Eric Badger added a comment - Jason Lowe , since HADOOP-13709 was committed, we can add in the localizer-specific piece. Here's a patch that attempts to shutdown the thread pool threads, kills all subprocesses, and then waits for the thread pool threads to terminate. If they do not terminate in 10 seconds, then we will call System.exit() to force the shutdown hook to be invoked, which will call destroyAllProcesses() again to make sure that all of the spawned subprocesses have been killed.
          Hide
          ebadger Eric Badger added a comment -

          Given the comments on HADOOP-13709, we will add the process bookkeeping and destroying methods in that JIRA. We can add in the Localizer-specific shutdown hook in this JIRA. This way we won't force unexpected behavior on those who assume that processes would still be running after shutdown (e.g. work-preserving restart).

          Show
          ebadger Eric Badger added a comment - Given the comments on HADOOP-13709 , we will add the process bookkeeping and destroying methods in that JIRA. We can add in the Localizer-specific shutdown hook in this JIRA. This way we won't force unexpected behavior on those who assume that processes would still be running after shutdown (e.g. work-preserving restart).
          Hide
          jlowe Jason Lowe added a comment -

          Given the direction HADOOP-13709 is taking, I don't think this change is appropriate. Since the executor threads executing commands via Shell will not be interruptible in practice, this change will simply make most localizers hang for an extra 10 seconds if there is an outstanding shell process. It would be faster to do the existing shutdownNow call and let the shell cleanup shutdown hook kill the underlying processes.

          One thought to make this more robust: if the localizer could explicitly invoke the "kill all active shell processes" hook manually, then it could do more of the cleanup itself like this:

            exec.shutdown();
            kill_all_active_shells_here();
            exec.awaitTermination(10, TimeUnit.SECONDS);
          

          So either this JIRA should be marked as a duplicate of HADOOP-13709 or updated per above.

          Show
          jlowe Jason Lowe added a comment - Given the direction HADOOP-13709 is taking, I don't think this change is appropriate. Since the executor threads executing commands via Shell will not be interruptible in practice, this change will simply make most localizers hang for an extra 10 seconds if there is an outstanding shell process. It would be faster to do the existing shutdownNow call and let the shell cleanup shutdown hook kill the underlying processes. One thought to make this more robust: if the localizer could explicitly invoke the "kill all active shell processes" hook manually, then it could do more of the cleanup itself like this: exec.shutdown(); kill_all_active_shells_here(); exec.awaitTermination(10, TimeUnit.SECONDS); So either this JIRA should be marked as a duplicate of HADOOP-13709 or updated per above.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 51s trunk passed
          +1 compile 0m 37s trunk passed
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 38s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 0m 55s trunk passed
          +1 javadoc 0m 23s trunk passed
          +1 mvninstall 0m 28s the patch passed
          +1 compile 0m 31s the patch passed
          +1 javac 0m 31s the patch passed
          +1 checkstyle 0m 18s the patch passed
          +1 mvnsite 0m 32s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 55s the patch passed
          +1 javadoc 0m 18s the patch passed
          +1 unit 15m 36s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          32m 27s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832726/YARN-5641.004.patch
          JIRA Issue YARN-5641
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 74e94e712634 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2fb392a
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13351/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13351/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 51s trunk passed +1 compile 0m 37s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 55s trunk passed +1 javadoc 0m 23s trunk passed +1 mvninstall 0m 28s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 55s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 15m 36s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 32m 27s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832726/YARN-5641.004.patch JIRA Issue YARN-5641 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 74e94e712634 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2fb392a Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13351/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13351/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Attaching a new patch that includes only the YARN portion of the fix.

          Show
          ebadger Eric Badger added a comment - Attaching a new patch that includes only the YARN portion of the fix.
          Hide
          ebadger Eric Badger added a comment - - edited

          Linking the hadoop-common portion of this fix as Jason Lowe suggested.

          Show
          ebadger Eric Badger added a comment - - edited Linking the hadoop-common portion of this fix as Jason Lowe suggested.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for the patch, Eric!

          This has primarily become a change to the Shell in hadoop-common to fix a problem with being able to interrupt it and kill the subprocess. Therefore I think that change needs to be a JIRA in HADOOP rather than YARN along with a better headline description. I'd move this one over to that project myself, but there's the YARN change that is also needed. We could either move this or file a JIRA in HADOOP that this depends upon. My preference is slightly towards the latter but deferring the choice to you.

          It would be good to have the error thread treated consistently, since now stderr is handled with a raw thread and stdout is handled with an executor service.

          Speaking of executor service, why are we using a scheduled thread executor service? We don't need the "scheduled" part of that service since we're just using the submit method.

          The OutOfMemoryError handling for the error thread isn't being applied to the new stdout handling and should be for consistency.

          The error handling of the future is silently eating any exception thrown whose cause isn't an IOException or a RuntimeException. IMHO if the cause is an IOException we should throw that, otherwise we should wrap it in a RuntimeException.

          Show
          jlowe Jason Lowe added a comment - Thanks for the patch, Eric! This has primarily become a change to the Shell in hadoop-common to fix a problem with being able to interrupt it and kill the subprocess. Therefore I think that change needs to be a JIRA in HADOOP rather than YARN along with a better headline description. I'd move this one over to that project myself, but there's the YARN change that is also needed. We could either move this or file a JIRA in HADOOP that this depends upon. My preference is slightly towards the latter but deferring the choice to you. It would be good to have the error thread treated consistently, since now stderr is handled with a raw thread and stdout is handled with an executor service. Speaking of executor service, why are we using a scheduled thread executor service? We don't need the "scheduled" part of that service since we're just using the submit method. The OutOfMemoryError handling for the error thread isn't being applied to the new stdout handling and should be for consistency. The error handling of the future is silently eating any exception thrown whose cause isn't an IOException or a RuntimeException. IMHO if the cause is an IOException we should throw that, otherwise we should wrap it in a RuntimeException.
          Hide
          ebadger Eric Badger added a comment -

          The test failure is unrelated to this patch and passes for me locally. Naganarasimha G R, Jason Lowe, could you review the most recent patch? Thanks!

          Show
          ebadger Eric Badger added a comment - The test failure is unrelated to this patch and passes for me locally. Naganarasimha G R , Jason Lowe , could you review the most recent patch? Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 42s trunk passed
          +1 compile 7m 0s trunk passed
          +1 checkstyle 1m 27s trunk passed
          +1 mvnsite 1m 24s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 2m 8s trunk passed
          +1 javadoc 1m 3s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 4s the patch passed
          +1 compile 7m 34s the patch passed
          +1 javac 7m 34s the patch passed
          -1 checkstyle 1m 30s root: The patch generated 2 new + 76 unchanged - 0 fixed = 78 total (was 76)
          +1 mvnsite 1m 24s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 28s the patch passed
          +1 javadoc 1m 4s the patch passed
          -1 unit 7m 31s hadoop-common in the patch failed.
          +1 unit 14m 47s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          59m 50s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829279/YARN-5641.003.patch
          JIRA Issue YARN-5641
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4b8c40b5ef60 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7558dbb
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13155/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13155/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13155/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13155/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13155/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 42s trunk passed +1 compile 7m 0s trunk passed +1 checkstyle 1m 27s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 2m 8s trunk passed +1 javadoc 1m 3s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 4s the patch passed +1 compile 7m 34s the patch passed +1 javac 7m 34s the patch passed -1 checkstyle 1m 30s root: The patch generated 2 new + 76 unchanged - 0 fixed = 78 total (was 76) +1 mvnsite 1m 24s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 28s the patch passed +1 javadoc 1m 4s the patch passed -1 unit 7m 31s hadoop-common in the patch failed. +1 unit 14m 47s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 59m 50s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829279/YARN-5641.003.patch JIRA Issue YARN-5641 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4b8c40b5ef60 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7558dbb Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13155/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13155/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13155/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13155/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13155/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Attaching new patch that adds in a unit test to make sure that the thread that invokes the ShellCommandExecutor.execute() method is now interruptible.

          Show
          ebadger Eric Badger added a comment - Attaching new patch that adds in a unit test to make sure that the thread that invokes the ShellCommandExecutor.execute() method is now interruptible.
          Hide
          ebadger Eric Badger added a comment -

          TestDNS and TestWebDelegationToken don't fail for me locally and are irrelevant to this patch.

          TestDefaultContainerExecutor was fixed just after this precommit build by YARN-5657.

          Show
          ebadger Eric Badger added a comment - TestDNS and TestWebDelegationToken don't fail for me locally and are irrelevant to this patch. TestDefaultContainerExecutor was fixed just after this precommit build by YARN-5657 .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 1s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 7m 41s trunk passed
          +1 compile 7m 33s trunk passed
          +1 checkstyle 1m 24s trunk passed
          +1 mvnsite 1m 22s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 2m 1s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 1s the patch passed
          +1 compile 6m 47s the patch passed
          +1 javac 6m 47s the patch passed
          -1 checkstyle 1m 27s root: The patch generated 3 new + 58 unchanged - 0 fixed = 61 total (was 58)
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 14s the patch passed
          +1 javadoc 1m 1s the patch passed
          -1 unit 15m 57s hadoop-common in the patch failed.
          -1 unit 14m 1s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          67m 41s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS
            hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor
          Timed out junit tests org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828911/YARN-5641.002.patch
          JIRA Issue YARN-5641
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b78286454caa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cd099a6
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13130/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13130/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13130/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13130/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-YARN-Build/13130/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13130/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13130/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 7m 41s trunk passed +1 compile 7m 33s trunk passed +1 checkstyle 1m 24s trunk passed +1 mvnsite 1m 22s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 2m 1s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 1s the patch passed +1 compile 6m 47s the patch passed +1 javac 6m 47s the patch passed -1 checkstyle 1m 27s root: The patch generated 3 new + 58 unchanged - 0 fixed = 61 total (was 58) +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 14s the patch passed +1 javadoc 1m 1s the patch passed -1 unit 15m 57s hadoop-common in the patch failed. -1 unit 14m 1s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 67m 41s Reason Tests Failed junit tests hadoop.net.TestDNS   hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor Timed out junit tests org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828911/YARN-5641.002.patch JIRA Issue YARN-5641 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b78286454caa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cd099a6 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13130/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13130/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13130/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13130/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-YARN-Build/13130/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13130/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13130/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          This new patch should fix most, if not all, of the test failures. The tests were failing because they were unable to get the IOException that came from parseExecResult(). Since it was in a child thread, the parent thread didn't have an exception to throw up the stack to the test.

          It would be cleaner to also include the errThread as an ExecutorService in the thread pool, but I've left it as its own standalone thread for now to keep the size of the patch down. If there is agreement, I can unify them into the same thread pool for consistency.

          Show
          ebadger Eric Badger added a comment - This new patch should fix most, if not all, of the test failures. The tests were failing because they were unable to get the IOException that came from parseExecResult(). Since it was in a child thread, the parent thread didn't have an exception to throw up the stack to the test. It would be cleaner to also include the errThread as an ExecutorService in the thread pool, but I've left it as its own standalone thread for now to keep the size of the patch down. If there is agreement, I can unify them into the same thread pool for consistency.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 23s Maven dependency ordering for branch
          +1 mvninstall 7m 10s trunk passed
          +1 compile 7m 13s trunk passed
          +1 checkstyle 1m 27s trunk passed
          +1 mvnsite 1m 27s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 2m 2s trunk passed
          +1 javadoc 1m 6s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 4s the patch passed
          +1 compile 7m 14s the patch passed
          +1 javac 7m 14s the patch passed
          +1 checkstyle 1m 24s the patch passed
          +1 mvnsite 1m 22s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 21s the patch passed
          +1 javadoc 1m 9s the patch passed
          -1 unit 7m 54s hadoop-common in the patch failed.
          -1 unit 14m 21s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          61m 20s



          Reason Tests
          Failed junit tests hadoop.fs.TestSymlinkLocalFSFileSystem
            hadoop.fs.TestStat
            hadoop.fs.TestSymlinkLocalFSFileContext
            hadoop.util.TestNodeHealthScriptRunner
            hadoop.fs.TestLocalFileSystem
            hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828851/YARN-5641.001.patch
          JIRA Issue YARN-5641
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8c6ec7f1a068 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b09a03c
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13122/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13122/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13122/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-YARN-Build/13122/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13122/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13122/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 23s Maven dependency ordering for branch +1 mvninstall 7m 10s trunk passed +1 compile 7m 13s trunk passed +1 checkstyle 1m 27s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 2m 2s trunk passed +1 javadoc 1m 6s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 4s the patch passed +1 compile 7m 14s the patch passed +1 javac 7m 14s the patch passed +1 checkstyle 1m 24s the patch passed +1 mvnsite 1m 22s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 21s the patch passed +1 javadoc 1m 9s the patch passed -1 unit 7m 54s hadoop-common in the patch failed. -1 unit 14m 21s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 61m 20s Reason Tests Failed junit tests hadoop.fs.TestSymlinkLocalFSFileSystem   hadoop.fs.TestStat   hadoop.fs.TestSymlinkLocalFSFileContext   hadoop.util.TestNodeHealthScriptRunner   hadoop.fs.TestLocalFileSystem   hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828851/YARN-5641.001.patch JIRA Issue YARN-5641 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8c6ec7f1a068 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b09a03c Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/13122/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13122/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13122/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-YARN-Build/13122/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13122/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13122/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Attaching a patch that creates a separate thread to read the input stream instead of doing it in the main localizer thread. This way the main thread can get interrupted during process.waitFor() and destroy the child threads in the finally block. I tested this using a sleep command instead of the tar command and it killed the sleep command when the localizer was killed.

          Show
          ebadger Eric Badger added a comment - Attaching a patch that creates a separate thread to read the input stream instead of doing it in the main localizer thread. This way the main thread can get interrupted during process.waitFor() and destroy the child threads in the finally block. I tested this using a sleep command instead of the tar command and it killed the sleep command when the localizer was killed.
          Hide
          ebadger Eric Badger added a comment -

          Jason Lowe and I worked on this for some time yesterday and killing the spawned untar shell process is proving to be very difficult. The localizer spawns up the untar shell thread, which invokes a shell exec untar command. Once the container is killed, the next time the localizer heartbeats to the NM, it will be instructed to die. Inside of the 'die' codepath, the localizer interrupts all of its spawned threads using the cancel() method. However, the untar thread is stuck inside of file I/O waiting to parse the result of the shell execution and is uninterruptible. The untar thread won't get the InterruptedException until it is finished, and so we cannot kill it or the untar shell exec before it completes. We can have the localizer process wait for the untar thread to end via awaitTermination() (currently it only uses shutdownNow()), but it won't return until untar finishes on its own, since shutdown() won't have any effect with interrupting the untar thread.

          I tested this by replacing the untar shell command with a sleep command so that there would be no worry about the untar actually finishing. The container was killed and instructed to die after the subsequent NM heartbeat. Then it attempted to shutdown all of its threads, but the untar thread would sit in readBytes instead of getting the InterruptedException. Below is the stack trace of the untar thread just after the localizer calls shutdown(). It never gets the InterruptedException and sits in this stack trace until awaitTermination hits its timeout and the localizer kills the JVM. Since we never catch the InterruptedException, we are unable to destroy the untar shell process and it continues to run after the localizer and untar thread are killed (it became owned by init).

          "ContainerLocalizer Downloader" #19 prio=5 os_prio=0 tid=0x00007f4315169800 nid=0x1530 runnable [0x00007f42f5217000]
             java.lang.Thread.State: RUNNABLE
          	at java.io.FileInputStream.readBytes(Native Method)
          	at java.io.FileInputStream.read(FileInputStream.java:255)
          	at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
          	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
          	- locked <0x000000076f4fca28> (a java.lang.UNIXProcess$ProcessPipeInputStream)
          	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
          	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
          	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
          	- locked <0x000000076f506cf8> (a java.io.InputStreamReader)
          	at java.io.InputStreamReader.read(InputStreamReader.java:184)
          	at java.io.BufferedReader.fill(BufferedReader.java:161)
          	at java.io.BufferedReader.read1(BufferedReader.java:212)
          	at java.io.BufferedReader.read(BufferedReader.java:286)
          	- locked <0x000000076f506cf8> (a java.io.InputStreamReader)
          	at org.apache.hadoop.util.Shell$ShellCommandExecutor.parseExecResult(Shell.java:786)
          	at org.apache.hadoop.util.Shell.runCommand(Shell.java:568)
          	at org.apache.hadoop.util.Shell.run(Shell.java:479)
          	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
          	at org.apache.hadoop.fs.FileUtil.unTarUsingTar(FileUtil.java:682)
          	at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:651)
          	at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:283)
          	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:364)
          	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          	at java.lang.Thread.run(Thread.java:745)
          
          Show
          ebadger Eric Badger added a comment - Jason Lowe and I worked on this for some time yesterday and killing the spawned untar shell process is proving to be very difficult. The localizer spawns up the untar shell thread, which invokes a shell exec untar command. Once the container is killed, the next time the localizer heartbeats to the NM, it will be instructed to die. Inside of the 'die' codepath, the localizer interrupts all of its spawned threads using the cancel() method. However, the untar thread is stuck inside of file I/O waiting to parse the result of the shell execution and is uninterruptible. The untar thread won't get the InterruptedException until it is finished, and so we cannot kill it or the untar shell exec before it completes. We can have the localizer process wait for the untar thread to end via awaitTermination() (currently it only uses shutdownNow()), but it won't return until untar finishes on its own, since shutdown() won't have any effect with interrupting the untar thread. I tested this by replacing the untar shell command with a sleep command so that there would be no worry about the untar actually finishing. The container was killed and instructed to die after the subsequent NM heartbeat. Then it attempted to shutdown all of its threads, but the untar thread would sit in readBytes instead of getting the InterruptedException. Below is the stack trace of the untar thread just after the localizer calls shutdown(). It never gets the InterruptedException and sits in this stack trace until awaitTermination hits its timeout and the localizer kills the JVM. Since we never catch the InterruptedException, we are unable to destroy the untar shell process and it continues to run after the localizer and untar thread are killed (it became owned by init). "ContainerLocalizer Downloader" #19 prio=5 os_prio=0 tid=0x00007f4315169800 nid=0x1530 runnable [0x00007f42f5217000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:255) at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked <0x000000076f4fca28> (a java.lang.UNIXProcess$ProcessPipeInputStream) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) - locked <0x000000076f506cf8> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.read1(BufferedReader.java:212) at java.io.BufferedReader.read(BufferedReader.java:286) - locked <0x000000076f506cf8> (a java.io.InputStreamReader) at org.apache.hadoop.util.Shell$ShellCommandExecutor.parseExecResult(Shell.java:786) at org.apache.hadoop.util.Shell.runCommand(Shell.java:568) at org.apache.hadoop.util.Shell.run(Shell.java:479) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773) at org.apache.hadoop.fs.FileUtil.unTarUsingTar(FileUtil.java:682) at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:651) at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:283) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:364) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
          Hide
          ebadger Eric Badger added a comment -

          The problem appears to be a race between FileUtil.java:unTarUsingTar finishing and the DeletionService deleting the localized files. If the container is killed before the untar is finished, then it will immediately try and delete the localized directory. Since tar is running independently of the DeletionService, it will continue to untar even after the directory contents have been deleted. This is why we see localized files even when the DeletionService runs successfully.

          I believe that the fix is to kill the untar process when the localize process is killed. If we can't kill it, then we at least need to wait for it to finish so that we can clean up the entire contents of what it untarred.

          Show
          ebadger Eric Badger added a comment - The problem appears to be a race between FileUtil.java:unTarUsingTar finishing and the DeletionService deleting the localized files. If the container is killed before the untar is finished, then it will immediately try and delete the localized directory. Since tar is running independently of the DeletionService, it will continue to untar even after the directory contents have been deleted. This is why we see localized files even when the DeletionService runs successfully. I believe that the fix is to kill the untar process when the localize process is killed. If we can't kill it, then we at least need to wait for it to finish so that we can clean up the entire contents of what it untarred.
          Hide
          ebadger Eric Badger added a comment -

          Naganarasimha G R, I see the issue in 2.7. I suspect that it is also an issue in 2.6 as well. I'm not aware of any fix in 2.8 or above, but it's possible that this has been fixed.

          Show
          ebadger Eric Badger added a comment - Naganarasimha G R , I see the issue in 2.7. I suspect that it is also an issue in 2.6 as well. I'm not aware of any fix in 2.8 or above, but it's possible that this has been fixed.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          which version?

          Show
          Naganarasimha Naganarasimha G R added a comment - which version?
          Hide
          ebadger Eric Badger added a comment -

          Here is an example log of the failure. The localizer attempts to clean up its localized directories and fails, thus leaving the directories behind.

          2016-09-09 00:11:10,483 [LocalizerRunner for container_e01_1469258802232_9766291_01_000505] INFO localizer.ResourceLocalizationService: Localizer failed
          java.io.IOException: java.lang.InterruptedException
                  at org.apache.hadoop.util.Shell.runCommand(Shell.java:565)
                  at org.apache.hadoop.util.Shell.run(Shell.java:459)
                  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:747)
                  at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:237)
                  at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1113)
          2016-09-09 00:11:10,484 [DeletionService #0] INFO nodemanager.LinuxContainerExecutor: Deleting path : /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/13
          2016-09-09 00:11:10,486 [DeletionService #0] INFO nodemanager.LinuxContainerExecutor: Deleting path : /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/13_tmp
          2016-09-09 00:11:10,486 [DeletionService #0] INFO nodemanager.LinuxContainerExecutor: Deleting path : /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/12
          2016-09-09 00:11:10,486 [DeletionService #0] INFO nodemanager.LinuxContainerExecutor: Deleting path : /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/12_tmp
          2016-09-09 00:11:10,491 [DeletionService #0] ERROR nodemanager.LinuxContainerExecutor: DeleteAsUser for /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/13 /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/13_tmp /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/12 /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/12_tmp returned with exit code: 255
          ExitCodeException exitCode=255: 
                  at org.apache.hadoop.util.Shell.runCommand(Shell.java:562)
                  at org.apache.hadoop.util.Shell.run(Shell.java:459)
                  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:747)
                  at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.deleteAsUser(LinuxContainerExecutor.java:424)
                  at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:295)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                  at java.lang.Thread.run(Thread.java:745)
          2016-09-09 00:11:10,491 [DeletionService #0] ERROR nodemanager.LinuxContainerExecutor: Output from LinuxContainerExecutor's deleteAsUser follows:
          2016-09-09 00:11:10,491 [DeletionService #0] INFO nodemanager.ContainerExecutor: main : command provided 3
          2016-09-09 00:11:10,491 [DeletionService #0] INFO nodemanager.ContainerExecutor: main : user is foobar
          2016-09-09 00:11:10,491 [DeletionService #0] INFO nodemanager.ContainerExecutor: main : requested yarn user is foobar
          2016-09-09 00:11:10,491 [DeletionService #0] INFO nodemanager.ContainerExecutor: rmdir of the/tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/12/ failed - Permission denied
          
          Show
          ebadger Eric Badger added a comment - Here is an example log of the failure. The localizer attempts to clean up its localized directories and fails, thus leaving the directories behind. 2016-09-09 00:11:10,483 [LocalizerRunner for container_e01_1469258802232_9766291_01_000505] INFO localizer.ResourceLocalizationService: Localizer failed java.io.IOException: java.lang.InterruptedException at org.apache.hadoop.util.Shell.runCommand(Shell.java:565) at org.apache.hadoop.util.Shell.run(Shell.java:459) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:747) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1113) 2016-09-09 00:11:10,484 [DeletionService #0] INFO nodemanager.LinuxContainerExecutor: Deleting path : /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/13 2016-09-09 00:11:10,486 [DeletionService #0] INFO nodemanager.LinuxContainerExecutor: Deleting path : /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/13_tmp 2016-09-09 00:11:10,486 [DeletionService #0] INFO nodemanager.LinuxContainerExecutor: Deleting path : /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/12 2016-09-09 00:11:10,486 [DeletionService #0] INFO nodemanager.LinuxContainerExecutor: Deleting path : /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/12_tmp 2016-09-09 00:11:10,491 [DeletionService #0] ERROR nodemanager.LinuxContainerExecutor: DeleteAsUser for /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/13 /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/13_tmp /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/12 /tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/12_tmp returned with exit code: 255 ExitCodeException exitCode=255: at org.apache.hadoop.util.Shell.runCommand(Shell.java:562) at org.apache.hadoop.util.Shell.run(Shell.java:459) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:747) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.deleteAsUser(LinuxContainerExecutor.java:424) at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:295) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-09-09 00:11:10,491 [DeletionService #0] ERROR nodemanager.LinuxContainerExecutor: Output from LinuxContainerExecutor's deleteAsUser follows: 2016-09-09 00:11:10,491 [DeletionService #0] INFO nodemanager.ContainerExecutor: main : command provided 3 2016-09-09 00:11:10,491 [DeletionService #0] INFO nodemanager.ContainerExecutor: main : user is foobar 2016-09-09 00:11:10,491 [DeletionService #0] INFO nodemanager.ContainerExecutor: main : requested yarn user is foobar 2016-09-09 00:11:10,491 [DeletionService #0] INFO nodemanager.ContainerExecutor: rmdir of the/tmp/yarn-local/usercache/foobar/appcache/application_1469258802232_9766291/filecache/12/ failed - Permission denied

            People

            • Assignee:
              ebadger Eric Badger
              Reporter:
              ebadger Eric Badger
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development