[YARN-6156] AM blacklisting to consider node label partition

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-alpha4
• Hadoop Flags: Reviewed
• Assignee: Bibin Chundatt
• Reporter: Bibin Chundatt
• Created: 07/Feb/17 12:40
• Updated: 15/Feb/17 09:30
• Resolved: 15/Feb/17 09:30

Description

As per the current implementation for AM blacklisting, disabling is based on complete cluster resource.
Incase of partitioned cluster though, nodes applicable for AM labels only should be considered as total nodes.

Additionally for unmanaged AM as AM blacklisting is not done as AM is not launched by RM, it is equivalent to AM blacklisting being disabled for this app. Hence we do not need to create a SimpleBlacklistManager object in RMAppImpl#createNewAttempt.
We can create DisabledBlacklistManager object instead.

    BlacklistManager currentAMBlacklist;
    if (currentAttempt != null) {
      currentAMBlacklist = currentAttempt.getAMBlacklist();
    } else {
      if (amBlacklistingEnabled) {
        currentAMBlacklist = new SimpleBlacklistManager(
            scheduler.getNumClusterNodes(), blacklistDisableThreshold);
      } else {
        currentAMBlacklist = new DisabledBlacklistManager();
      }
    }

Issue Links

• is related to: YARN-6148 NM node count reported to AM in Allocate Response should consider requested node label partitions.