Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3862

Nodemanager can appear to hang on shutdown due to lingering DeletionService threads

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.1
    • Fix Version/s: 0.23.2
    • Component/s: mrv2, nodemanager
    • Labels:
      None

      Description

      When a nodemanager attempts to shutdown cleanly, it's possible for it to appear to hang due to lingering DeletionService threads. This can occur when yarn.nodemanager.delete.debug-delay-sec is set to a relatively large value and one or more containers executes on the node shortly before the shutdown.

      The DeletionService is never calling setExecuteExistingDelayedTasksAfterShutdownPolicy() on the ScheduledThreadPoolExecutor, and it defaults to waiting for all scheduled tasks to complete before exiting.

      1. MAPREDUCE-3862.patch
        5 kB
        Jason Lowe
      2. MAPREDUCE-3862.patch
        4 kB
        Jason Lowe

        Issue Links

          Activity

          Hide
          Vinod Kumar Vavilapalli added a comment -

          Thanks, created MAPREDUCE-3888.

          Show
          Vinod Kumar Vavilapalli added a comment - Thanks, created MAPREDUCE-3888 .
          Hide
          Robert Joseph Evans added a comment -

          I do not know of an open ticket for this. I assumed that most components should be doing this anyways, because if the process crashes badly there is no guarantee that the files would be deleted, so I assumed that it should have been part of their initial design. If that is not the case then yes we should file a JIRA for it.

          Show
          Robert Joseph Evans added a comment - I do not know of an open ticket for this. I assumed that most components should be doing this anyways, because if the process crashes badly there is no guarantee that the files would be deleted, so I assumed that it should have been part of their initial design. If that is not the case then yes we should file a JIRA for it.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          we have to be able to handle starting up from an unclean shutdown, so files need to be deleted on startup as well. +1

          Yes, that is what happens in 1.0.* too. You know of an open ticket for this?

          Show
          Vinod Kumar Vavilapalli added a comment - we have to be able to handle starting up from an unclean shutdown, so files need to be deleted on startup as well. +1 Yes, that is what happens in 1.0.* too. You know of an open ticket for this?
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #994 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/994/)
          MAPREDUCE-3862 Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245781)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245781
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #994 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/994/ ) MAPREDUCE-3862 Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245781) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245781 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #200 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/200/)
          svn merge -c 1245781 from trunk to branch 0.23 FIXES MAPREDUCE-3862. Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245794)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245794
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #200 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/200/ ) svn merge -c 1245781 from trunk to branch 0.23 FIXES MAPREDUCE-3862 . Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245794) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245794 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #172 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/172/)
          svn merge -c 1245781 from trunk to branch 0.23 FIXES MAPREDUCE-3862. Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245794)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245794
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #172 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/172/ ) svn merge -c 1245781 from trunk to branch 0.23 FIXES MAPREDUCE-3862 . Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245794) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245794 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #959 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/959/)
          MAPREDUCE-3862 Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245781)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245781
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #959 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/959/ ) MAPREDUCE-3862 Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245781) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245781 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #571 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/571/)
          svn merge -c 1245781 from trunk to branch 0.23 FIXES MAPREDUCE-3862. Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245794)

          Result = ABORTED
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245794
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #571 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/571/ ) svn merge -c 1245781 from trunk to branch 0.23 FIXES MAPREDUCE-3862 . Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245794) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245794 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #568 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/568/)
          svn merge -c 1245781 from trunk to branch 0.23 FIXES MAPREDUCE-3862. Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245794)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245794
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #568 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/568/ ) svn merge -c 1245781 from trunk to branch 0.23 FIXES MAPREDUCE-3862 . Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245794) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245794 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #555 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/555/)
          svn merge -c 1245781 from trunk to branch 0.23 FIXES MAPREDUCE-3862. Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245794)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245794
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #555 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/555/ ) svn merge -c 1245781 from trunk to branch 0.23 FIXES MAPREDUCE-3862 . Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245794) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245794 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Hide
          Robert Joseph Evans added a comment -

          Thanks Jason I just committed this to trunk and branch 0.23

          Show
          Robert Joseph Evans added a comment - Thanks Jason I just committed this to trunk and branch 0.23
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1758 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1758/)
          MAPREDUCE-3862 Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245781)

          Result = ABORTED
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245781
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1758 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1758/ ) MAPREDUCE-3862 Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245781) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245781 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1746 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1746/)
          MAPREDUCE-3862 Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245781)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245781
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1746 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1746/ ) MAPREDUCE-3862 Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245781) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245781 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1820/)
          MAPREDUCE-3862 Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245781)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245781
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1820/ ) MAPREDUCE-3862 Nodemanager can appear to hang on shutdown due to lingering DeletionService threads (Jason Lowe via bobby) (Revision 1245781) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245781 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDeletionService.java
          Hide
          Robert Joseph Evans added a comment -

          I like the approach that this is taking. we have to be able to handle starting up from an unclean shutdown, so files need to be deleted on startup as well. +1 for the patch.

          Show
          Robert Joseph Evans added a comment - I like the approach that this is taking. we have to be able to handle starting up from an unclean shutdown, so files need to be deleted on startup as well. +1 for the patch.
          Hide
          Jason Lowe added a comment -

          Note that with this patch the DeletionService can leave some scheduled files undeleted to avoid long hangs at shutdown. A couple of alternatives:

          • Implement a customized ScheduledThreadPoolExecutor that executes all scheduled tasks immediately upon shutdown rather than waiting the specified delays for each scheduled task. This could still lead to long shutdown times if there are directories scheduled to be deleted with tons of files.
          • Declare the existing behavior "as-intended" and note that NMs can take up to yarn.nodemanager.delete.debug-delay-sec seconds to finish shutting down. Would be helpful to log a useful message when waiting.
          Show
          Jason Lowe added a comment - Note that with this patch the DeletionService can leave some scheduled files undeleted to avoid long hangs at shutdown. A couple of alternatives: Implement a customized ScheduledThreadPoolExecutor that executes all scheduled tasks immediately upon shutdown rather than waiting the specified delays for each scheduled task. This could still lead to long shutdown times if there are directories scheduled to be deleted with tons of files. Declare the existing behavior "as-intended" and note that NMs can take up to yarn.nodemanager.delete.debug-delay-sec seconds to finish shutting down. Would be helpful to log a useful message when waiting.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12514725/MAPREDUCE-3862.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1870//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1870//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12514725/MAPREDUCE-3862.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1870//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1870//console This message is automatically generated.
          Hide
          Jason Lowe added a comment -

          Patch updated for unit test failures.

          Show
          Jason Lowe added a comment - Patch updated for unit test failures.
          Hide
          Jason Lowe added a comment -

          Canceling patch to investigate test failures.

          Show
          Jason Lowe added a comment - Canceling patch to investigate test failures.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12514708/MAPREDUCE-3862.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.yarn.server.nodemanager.TestDeletionService

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1869//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1869//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12514708/MAPREDUCE-3862.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.nodemanager.TestDeletionService +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1869//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1869//console This message is automatically generated.
          Hide
          Jason Lowe added a comment -

          Patch to call setExecuteExistingDelayedTasksAfterShutdownPolicy() on init and fallback to shutdownNow() if ScheduledThreadPoolExecutor.awaitTermination() fails.

          Show
          Jason Lowe added a comment - Patch to call setExecuteExistingDelayedTasksAfterShutdownPolicy() on init and fallback to shutdownNow() if ScheduledThreadPoolExecutor.awaitTermination() fails.
          Hide
          Jason Lowe added a comment -

          DeletionService has the following code which implies we don't want to wait too long for the shutdown to complete:

            public void stop() {
              sched.shutdown();
              try {
                sched.awaitTermination(10, SECONDS);
              } catch (InterruptedException e) {
                sched.shutdownNow();
              }
              super.stop();
            }
          

          However the code never checks the result from awaitTermination(), and we can end up trying to continue the shutdown process with the thread pool still active.

          Show
          Jason Lowe added a comment - DeletionService has the following code which implies we don't want to wait too long for the shutdown to complete: public void stop() { sched.shutdown(); try { sched.awaitTermination(10, SECONDS); } catch (InterruptedException e) { sched.shutdownNow(); } super .stop(); } However the code never checks the result from awaitTermination() , and we can end up trying to continue the shutdown process with the thread pool still active.

            People

            • Assignee:
              Jason Lowe
              Reporter:
              Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development