Hadoop YARN
  1. Hadoop YARN
  2. YARN-688

Containers not cleaned up when NM received SHUTDOWN event from NodeStatusUpdater

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0-beta
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, both SHUTDOWN event from nodeStatusUpdater and CleanupContainers event happens to be on the same dispatcher thread, CleanupContainers Event will not be processed until SHUTDOWN event is processed. see similar problem on YARN-495.
      On normal NM shutdown, this is not a problem since normal stop happens on shutdownHook thread.

      1. YARN-688.1.patch
        6 kB
        Jian He
      2. YARN-688.2.patch
        8 kB
        Jian He
      3. YARN-688.3.patch
        8 kB
        Jian He
      4. YARN-688.4.patch
        8 kB
        Jian He
      5. YARN-688.5.patch
        9 kB
        Jian He
      6. YARN-688.6.patch
        11 kB
        Jian He
      7. YARN-688-2.0.5-alpha.patch
        3 kB
        Jian He

        Issue Links

          Activity

          Jian He created issue -
          Jian He made changes -
          Field Original Value New Value
          Link This issue relates to YARN-495 [ YARN-495 ]
          Hide
          Jian He added a comment -

          This patch basically creates a new thread on handling shutdown event from nodeStatusUpdater

          Show
          Jian He added a comment - This patch basically creates a new thread on handling shutdown event from nodeStatusUpdater
          Jian He made changes -
          Attachment YARN-688.1.patch [ 12583416 ]
          Jian He made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12583416/YARN-688.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/941//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/941//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12583416/YARN-688.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/941//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/941//console This message is automatically generated.
          Hide
          Omkar Vinit Joshi added a comment -

          + nodeStatusUpdater.getNodeStatusAndUpdateContainersInContext();

          why is this required?

          you might need to rebase patch based on YARN-617

          testNodeStatusUpdaterRetryAndNMShutdown - startContainer code might change after above patch.

          Show
          Omkar Vinit Joshi added a comment - + nodeStatusUpdater.getNodeStatusAndUpdateContainersInContext(); why is this required? you might need to rebase patch based on YARN-617 testNodeStatusUpdaterRetryAndNMShutdown - startContainer code might change after above patch.
          Hide
          Jian He added a comment -

          nodeStatusUpdater.getNodeStatusAndUpdateContainersInContext();

          this is required, since this method is removing the completed containers from the context

          Will rebase later.

          Show
          Jian He added a comment - nodeStatusUpdater.getNodeStatusAndUpdateContainersInContext(); this is required, since this method is removing the completed containers from the context Will rebase later.
          Hide
          Jian He added a comment -

          patch for 2.0.5-alpha for YARN-864

          Show
          Jian He added a comment - patch for 2.0.5-alpha for YARN-864
          Jian He made changes -
          Attachment YARN-688-2.0.5-alpha.patch [ 12589128 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12589128/YARN-688-2.0.5-alpha.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1376//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12589128/YARN-688-2.0.5-alpha.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1376//console This message is automatically generated.
          Hide
          Jian He added a comment -

          rebased patch for latest trunk

          Show
          Jian He added a comment - rebased patch for latest trunk
          Jian He made changes -
          Attachment YARN-688.2.patch [ 12589137 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12589137/YARN-688.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1377//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/1377//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1377//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12589137/YARN-688.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1377//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/1377//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1377//console This message is automatically generated.
          Jian He made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jian He made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          The patch isn't applying any more. Can you please update? Tx.

          Show
          Vinod Kumar Vavilapalli added a comment - The patch isn't applying any more. Can you please update? Tx.
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is duplicated by YARN-864 [ YARN-864 ]
          Hide
          Jian He added a comment -

          Rebased against latest trunk

          Show
          Jian He added a comment - Rebased against latest trunk
          Jian He made changes -
          Attachment YARN-688.3.patch [ 12593804 ]
          Jian He made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12593804/YARN-688.3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1558//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1558//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12593804/YARN-688.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1558//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1558//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Seems like it needs update again after YARN-926.

          Show
          Vinod Kumar Vavilapalli added a comment - Seems like it needs update again after YARN-926 .
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Jian He added a comment -

          updated patch

          Show
          Jian He added a comment - updated patch
          Jian He made changes -
          Attachment YARN-688.4.patch [ 12594010 ]
          Jian He made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12594010/YARN-688.4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1575//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1575//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12594010/YARN-688.4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1575//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1575//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Mostly looks fine. One comment, the test-case isn't running as expected. On my machine, though we expect the container to be long running, it crashes. So we aren't verifying the correct thing. I was trying to make the test fail without code changes. which is when I ran into this. Can you take a look?

          Show
          Vinod Kumar Vavilapalli added a comment - Mostly looks fine. One comment, the test-case isn't running as expected. On my machine, though we expect the container to be long running, it crashes. So we aren't verifying the correct thing. I was trying to make the test fail without code changes. which is when I ran into this. Can you take a look?
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Jian He added a comment -

          Updated the test, the test will fail without NM changes and pass with changes.

          Show
          Jian He added a comment - Updated the test, the test will fail without NM changes and pass with changes.
          Jian He made changes -
          Attachment YARN-688.5.patch [ 12594044 ]
          Jian He made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12594044/YARN-688.5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1579//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1579//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12594044/YARN-688.5.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1579//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1579//console This message is automatically generated.
          Jian He made changes -
          Attachment YARN-688.6.patch [ 12594087 ]
          Jian He made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jian He made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Jian He added a comment -

          new patch fixed test container to be long running.

          Show
          Jian He added a comment - new patch fixed test container to be long running.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12594087/YARN-688.6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1584//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1584//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12594087/YARN-688.6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1584//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1584//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          +1. Checking this in..

          Show
          Vinod Kumar Vavilapalli added a comment - +1. Checking this in..
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Committed this to trunk, branch-2 and branch-2.1. Thanks Jian!

          Show
          Vinod Kumar Vavilapalli added a comment - Committed this to trunk, branch-2 and branch-2.1. Thanks Jian!
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.1.0-beta [ 12324029 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Jian He
              Reporter:
              Jian He
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development