Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5489

MR jobs hangs as it does not use the node-blacklisting feature in RM requests

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.2.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When RM restarted, if during restart one NM went bad (bad disk), NM got blacklisted by AM and RM keeps giving the containers on the same node even though AM doesn't want it there.

      Need to change AM to specifically blacklist node in the RM requests.

      1. MAPREDUCE-5489.1.patch
        17 kB
        Zhijie Shen
      2. MAPREDUCE-5489.2.patch
        17 kB
        Zhijie Shen

        Issue Links

          Activity

          Hide
          bikassaha Bikas Saha added a comment -

          how is this related to rm restart?

          Show
          bikassaha Bikas Saha added a comment - how is this related to rm restart?
          Hide
          zjshen Zhijie Shen added a comment -

          Bikas Saha, RM restarting is not the root cause. In contrast, the root cause is that MR AM wants to blacklist one node, but RM doesn't know it.

          Show
          zjshen Zhijie Shen added a comment - Bikas Saha , RM restarting is not the root cause. In contrast, the root cause is that MR AM wants to blacklist one node, but RM doesn't know it.
          Hide
          bikassaha Bikas Saha added a comment -

          Can you please update the title to make it clear.

          Show
          bikassaha Bikas Saha added a comment - Can you please update the title to make it clear.
          Hide
          yeshavora Yesha Vora added a comment -

          Title updated.

          Show
          yeshavora Yesha Vora added a comment - Title updated.
          Hide
          zjshen Zhijie Shen added a comment -

          I've created the patch to make AM send blacklist nodes to RM. Basically the logical is described as follows:

          1. Add blacklistAdditions and blacklistRemovals to remember the blacklisted nodes added or removed between two allocate calls. The two collections will be sent to RM in upcoming allocate call.

          2. Whenever a container fails on a host, the host will be blacklisted, and will add to blacklistAdditions if blacklist is not ignored.

          3. When changing from not ignoring blacklist to ignoring, we added all the blacklist nodes to blacklistRemovals.

          4. When changing from ignoring blacklist to not ignoring, we added all the blacklist nodes to blacklistAdditions.

          5. Switching between ignoring and not ignoring blacklist nodes will not effect until the upcoming allocate call, but anyway, it will effect eventually.

          Test cases have been modified test whether RM is aware of the blacklisted nodes.

          Show
          zjshen Zhijie Shen added a comment - I've created the patch to make AM send blacklist nodes to RM. Basically the logical is described as follows: 1. Add blacklistAdditions and blacklistRemovals to remember the blacklisted nodes added or removed between two allocate calls. The two collections will be sent to RM in upcoming allocate call. 2. Whenever a container fails on a host, the host will be blacklisted, and will add to blacklistAdditions if blacklist is not ignored. 3. When changing from not ignoring blacklist to ignoring, we added all the blacklist nodes to blacklistRemovals. 4. When changing from ignoring blacklist to not ignoring, we added all the blacklist nodes to blacklistAdditions. 5. Switching between ignoring and not ignoring blacklist nodes will not effect until the upcoming allocate call, but anyway, it will effect eventually. Test cases have been modified test whether RM is aware of the blacklisted nodes.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12606496/MAPREDUCE-5489.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4080//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4080//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12606496/MAPREDUCE-5489.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4080//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4080//console This message is automatically generated.
          Hide
          bikassaha Bikas Saha added a comment -

          By the time we clear, could new entries have been added to the sets (and which will be lost upon clear()) ?

               release.clear();
          +
          +    if (blacklistAdditions.size() > 0 || blacklistRemovals.size() > 0) {
          +      LOG.info("Update the blacklist for " + applicationId +
          +          ": blacklistAdditions=" + blacklistAdditions.size() +
          +          " blacklistRemovals=" +  blacklistRemovals.size());
          +    }
          +    blacklistAdditions.clear();
          +    blacklistRemovals.clear();
          

          I think the logic should be to not blacklist additional nodes if the threshold is crossed. Instead the logic in the patch (and probably before it too) seems to be to completely ignore all blacklisted nodes if the threshold is crossed. That seems incorrect.
          If we choose to not blacklist additional new nodes then we should stop sending blacklist additions to the RM once threshold is reached. ie when the ignore flag becomes true.

          +          // notify RM to ignore all the blacklisted nodes
          +          blacklistAdditions.clear();
          +          blacklistRemovals.addAll(blacklistedNodes);
                   }
                 } else {
                   if (ignoreBlacklisting.compareAndSet(true, false)) {
                     LOG.info("Ignore blacklisting set to false. Known: " + clusterNmCount
                         + ", Blacklisted: " + blacklistedNodeCount + ", " + val + "%");
          +          // notify RM of all the blacklisted nodes
          +          blacklistAdditions.add
          

          Will the change be simpler if whenever the blacklisted nodes change then the entire list is sent to the RM. Additionally RM is not updated if the ignore flag is set.

          Show
          bikassaha Bikas Saha added a comment - By the time we clear, could new entries have been added to the sets (and which will be lost upon clear()) ? release.clear(); + + if (blacklistAdditions.size() > 0 || blacklistRemovals.size() > 0) { + LOG.info( "Update the blacklist for " + applicationId + + ": blacklistAdditions=" + blacklistAdditions.size() + + " blacklistRemovals=" + blacklistRemovals.size()); + } + blacklistAdditions.clear(); + blacklistRemovals.clear(); I think the logic should be to not blacklist additional nodes if the threshold is crossed. Instead the logic in the patch (and probably before it too) seems to be to completely ignore all blacklisted nodes if the threshold is crossed. That seems incorrect. If we choose to not blacklist additional new nodes then we should stop sending blacklist additions to the RM once threshold is reached. ie when the ignore flag becomes true. + // notify RM to ignore all the blacklisted nodes + blacklistAdditions.clear(); + blacklistRemovals.addAll(blacklistedNodes); } } else { if (ignoreBlacklisting.compareAndSet( true , false )) { LOG.info( "Ignore blacklisting set to false . Known: " + clusterNmCount + ", Blacklisted: " + blacklistedNodeCount + ", " + val + "%" ); + // notify RM of all the blacklisted nodes + blacklistAdditions.add Will the change be simpler if whenever the blacklisted nodes change then the entire list is sent to the RM. Additionally RM is not updated if the ignore flag is set.
          Hide
          zjshen Zhijie Shen added a comment -

          By the time we clear, could new entries have been added to the sets (and which will be lost upon clear()) ?

          Seems to be impossible, because blacklistAdditions an blacklistedRemovals are only referred in two synchronized methods: heartbeat() and handleEvent()

          I think the logic should be to not blacklist additional nodes if the threshold is crossed.

          Yes, if the threshold is reached, the blacklist will be be checked for the following assigned containers. That's why we need to notify the RM to remove the blacklist. Otherwise, the RM will continue to avoid assigning the containers on the blacklisted nodes to the AM; therefore, the nodes will actually still blacklisted.

          Will the change be simpler if whenever the blacklisted nodes change then the entire list is sent to the RM.

          We need to wait until next heatbeat to notify RM, such that we need to remember all the changes that's why blacklistAdditions is there to track the increased blacklisted nodes.

          Show
          zjshen Zhijie Shen added a comment - By the time we clear, could new entries have been added to the sets (and which will be lost upon clear()) ? Seems to be impossible, because blacklistAdditions an blacklistedRemovals are only referred in two synchronized methods: heartbeat() and handleEvent() I think the logic should be to not blacklist additional nodes if the threshold is crossed. Yes, if the threshold is reached, the blacklist will be be checked for the following assigned containers. That's why we need to notify the RM to remove the blacklist. Otherwise, the RM will continue to avoid assigning the containers on the blacklisted nodes to the AM; therefore, the nodes will actually still blacklisted. Will the change be simpler if whenever the blacklisted nodes change then the entire list is sent to the RM. We need to wait until next heatbeat to notify RM, such that we need to remember all the changes that's why blacklistAdditions is there to track the increased blacklisted nodes.
          Hide
          bikassaha Bikas Saha added a comment -

          Typos
          + assertBlackListAddtionsAndRemovals
          + // Because makeRemoteRequest will not be ware of it until next call

          In general the policy of ignoring all blacklisted nodes after the threshold is reached seems incorrect. Lets create a follow up jira to fix that.

          Show
          bikassaha Bikas Saha added a comment - Typos + assertBlackListAddtionsAndRemovals + // Because makeRemoteRequest will not be ware of it until next call In general the policy of ignoring all blacklisted nodes after the threshold is reached seems incorrect. Lets create a follow up jira to fix that.
          Hide
          zjshen Zhijie Shen added a comment -

          Fixed the typos in the newest patch.

          Show
          zjshen Zhijie Shen added a comment - Fixed the typos in the newest patch.
          Hide
          zjshen Zhijie Shen added a comment -

          In general the policy of ignoring all blacklisted nodes after the threshold is reached seems incorrect. Lets create a follow up jira to fix that.

          Filed MAPREDUCE-5559 to track the issue

          Show
          zjshen Zhijie Shen added a comment - In general the policy of ignoring all blacklisted nodes after the threshold is reached seems incorrect. Lets create a follow up jira to fix that. Filed MAPREDUCE-5559 to track the issue
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12606634/MAPREDUCE-5489.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4081//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4081//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12606634/MAPREDUCE-5489.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4081//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4081//console This message is automatically generated.
          Hide
          bikassaha Bikas Saha added a comment -

          Looks good +1.

          Show
          bikassaha Bikas Saha added a comment - Looks good +1.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #4527 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4527/)
          MAPREDUCE-5489. MR jobs hangs as it does not use the node-blacklisting feature in RM requests (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529005)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4527 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4527/ ) MAPREDUCE-5489 . MR jobs hangs as it does not use the node-blacklisting feature in RM requests (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529005 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #352 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/352/)
          MAPREDUCE-5489. MR jobs hangs as it does not use the node-blacklisting feature in RM requests (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529005)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #352 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/352/ ) MAPREDUCE-5489 . MR jobs hangs as it does not use the node-blacklisting feature in RM requests (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529005 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1542 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1542/)
          MAPREDUCE-5489. MR jobs hangs as it does not use the node-blacklisting feature in RM requests (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529005)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1542 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1542/ ) MAPREDUCE-5489 . MR jobs hangs as it does not use the node-blacklisting feature in RM requests (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529005 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1568 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1568/)
          MAPREDUCE-5489. MR jobs hangs as it does not use the node-blacklisting feature in RM requests (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529005)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1568 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1568/ ) MAPREDUCE-5489 . MR jobs hangs as it does not use the node-blacklisting feature in RM requests (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529005 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing old tickets that are already shipped in a release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing old tickets that are already shipped in a release.

            People

            • Assignee:
              zjshen Zhijie Shen
              Reporter:
              yeshavora Yesha Vora
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development