My proposal is:

For now, RM CapacityScheduler UI looks like

+ root [==========================] 50% used
  + a  [=======================================] 75% used
    - a1 [=================]  30% used
      ---------------------------------------------
      |          Queue Metrics Table               |
      |--------------------------------------------|
      |       metrics1       |            value1   |
      |       metrics2       |            value2   |
      |       metrics3       |            value3   |
      |       metrics4       |            value4   |
      ----------------------------------------------
  + b [...]
  + c [...]

We can add one more hierarchy above queue's hierarchy, which are labels can be accessed and/or labels are being used by the queue, which can looks like

+ label_x  [=====================] 30% used
	+ root [=====================] 30% used
	  + a  [=======================================] 75% used
	    + a1 [=================]  30% used
	      ---------------------------------------------
	      |          Queue Metrics Table (For label_x) |
	      |--------------------------------------------|
	      |       metrics1       |            value1   |
	      |       metrics2       |            value2   |
	      |       metrics3       |            value3   |
	      |       metrics4       |            value4   |
	      ----------------------------------------------
+ label_y
    + root [...]
    + ...
+ label_z
    + root [...]
    + ...
+ no_label
    + root [...]
    + ...

To make it backward compatible, when there's no label in the system, it will not show "label-bar", and root is still "root-queue".

Please feel free to share your ideas on this!

Hi Wangda,
Would like to work on this issue, hence have assigned to my name, if you have already started working on it please feel free to reassign.

It's yours . Looking forward your patch.

Thanks,

Thanks Tan, Wangda,
Regarding the approach to display i had few concerns :

• There will be some common queue metrics across the labels, wont it get repeated across for each label if a queue is mapped to multiple labels ?
• IIUC most of the queue Metrics might not be specific to a label, like Capacity, Absolute max capacity, Max apps, Max AM's per user etc... . Correct me if my understanding on this is wrong.
• Apart from the label specific queue metrics like (label capacity, label abs capacity,used) are there any new Label specific queue metrics you have in your mind ?
• would it be better to list like

  + root [=====================] 30% used
    + a  [=======================================] 75% used
      + a1 [=================]  30% used
  	       ---------------------------------------------
  	      |          Queue Metrics                     |
  	      |--------------------------------------------|
  	      |       metrics1       |            value1   |
  	      |       metrics2       |            value2   |
             ---------------------------------------------
  	      |          Active Users info  (yarn-3273)    |
  	      |--------------------------------------------|
  	      |       user1       |            info        |
  	      |       user2       |            info        |
             ---------------------------------------------
  	      |     Label Resource usage info              |
  	      |--------------------------------------------|
  	      | label_x  [=====================] 30% used  |
  	      | label_y  [================] 20% used       |
  	      ----------------------------------------------
      + a2 [=================]  30% used
      ...
  

• Also if required we can have seperate page (/in the labels page/append at the end of CS page) like :

  + label_x  [=====================] 30% used [Actual Resource - Used resource ]
  	+ root [=====================] 30% used [Actual Resource - Used resource ]
  	  + a  [=======================================] 75% used [Actual Resource - Used resource ]
  	    + a1 [=================]  30% used [Actual Resource - Used resource ]
  + label_y
      + root [...]
      + ...
  + label_z
      + root [...]
  

YARN-3273, has added more info to the CS page so we need to consider the size of page and its usability.
Please provide your thoughts on the same

Hi Naganarasimha G R,
Thanks for your comments,

There will be some common queue metrics across the labels, wont it get repeated across for each label if a queue is mapped to multiple labels?

Some common fields may get repeated (like absolute max capacity, etc.). I think repeat some of them is not a very big issue to me. I think we can show queue-label-metrics + queue-common-metrics for each queue-label

IIUC most of the queue Metrics might not be specific to a label, like Capacity, Absolute max capacity, Max apps, Max AM's per user etc... . Correct me if my understanding on this is wrong.

Yes, they're, but there're more parameters / metrics in queues for both label/queue, different labels under same queue can have different user-limit/capacity/maximum-capacity/max-am-resource, etc.). We also need to show them to users if possible

Apart from the label specific queue metrics like (label capacity, label abs capacity,used) are there any new Label specific queue metrics you have in your mind ?

I think above can answer your question.

would it be better to list like

If we have this view,
1) How you show label-specific metrics?
2) What's the "used-resource" in queue level means (used-resource make more sense when it's per-label).
3) How to check "label-wise" resource usage for parent queues.

Also if required we can have seperate page (/in the labels page/append at the end of CS page) like

I think my proposal is still a little more clear, we need to show label-wise metrics to user. And with that, user can clear understand resource usage for each partition (just check each label's usage. Also parent's label-wise usage can show as well.

For the active-user-info, we need some queue-user-by-label metrics as well, such as used-resource-by-user-and-label. Which can be placed to queue-label metrics table.

Thanks for the feedback Wangda Tan,

different labels under same queue can have different user-limit/capacity/maximum-capacity/max-am-resource, etc.

If this is the case then the approach which you specified makes sense but by "can" you mean currently its not there and in future it can come in ?

More than repeated info, other drawback i can see is suppose for particular label userlimit is not reached but as overall at queue level if the user has reached his limit it will be difficult for user to go through all labels and find out whether user has reached queue limit . Correct me if my understanding on this is wrong .

If this is the case then the approach which you specified makes sense but by "can" you mean currently its not there and in future it can come in ?

Some of them are already existed, like user-limit, and some of them are coming, like am-resource-percent.

Sorry I may not understand what's your question, user-limit and queue-limit are just two different limits regardless of node labels, sometimes user-limit higher and sometimes queue-limit higher. Could you explain what's your question (maybe by example)?

Thanks,

Since this patch tends to create a new RM web view, so I suggest to do it in a compatible way so that people will not get lost when using the new view:

• When label is not enabled in the cluster or there's only "default" label, it will not show the "label-hierarchy", so it will be very similar to what we have in old RM web UI
• All queue's capacities and usages (by partition) can be found in CSQueue.getQueueCapacities/getQueueResourceUsages.
• We may not need to consider show app-by-partition with this patch. Filter apps in a queue by partition needs add additional CS interface, which should be addressed in a separated patch.

Hi Tan, Wangda, Have uploaded WIP Patch, i have not done much of testing and one of the main reason being i am not able to get the hierarchy when multiple labels are present. Attaching the initial version of the patch which will help in providing feedback earlier. Hope some one can help me in understanding the foldable lists of hammock framework and review the modifications in CapacitySchedulerPage(line no 324 - 341) for it.
Also one more thing, i have tried to avoid displaying

• Labels which do not have any resource (no node mapping)
• Labels which are not accessible
• Sub queue hierarchies if the labels are not accessible to it.

Hi Naga,
Thanks for taking initiative for this, just tried to run the patch locally, looks great! Some comments:

1) Show partition=partition-name in every partition, if the partition is the NO_LABEL partition, show it's a YARN.DEFAULT.PARTITION.
2) I think it's better to show labels are not accessible, especially for the non-exclusive node label case, we can optimize this in future patch. To avoid people ask question like "where is my label"? This includes all existing "avoid displaying" items in your existing patch. But it's good to keep avoid showing "label" when there's no label in your cluster.
3) Showing partition of partition-specific queue metrics, they're:

• Used Capacity: 0.0%
• Absolute Used Capacity: 0.0%
• Absolute Capacity: 50.0%
• Absolute Max Capacity: 100.0%
• Configured Capacity: 50.0%
• Configured Max Capacity: 100.0%
  I suggest to add a (Partition=xxx) at the end of these metrics.

I attached queue hierarchy showing in my local cluster: https://issues.apache.org/jira/secure/attachment/12729256/Screen%20Shot%202015-04-29%20at%2011.42.17%20AM.png. It seems multi hierarchy works well in my environment.

Thanks Tan, Wangda, for reviewing and testing the patch.

partition=partition-name

Well i understand that in the later patches we are targetting it more as partition than labels, but in that case shall i modify the same in other locations of WEB like node labels page, in CS page shall i mark it as Accessible Partitions ?

But it's good to keep avoid showing "label" when there's no label in your cluster.

you mean if no node is mapped to cluster node label then not to show that Node Label ?

Showing partition of partition-specific queue metrics

you mean the existing names of metrics entries needs to be appended with (Partition=xxx) and not to show both right ?

It seems multi hierarchy works well in my environment.

Its great to hear its working fine, but it worked without any modifications to the patch ? If so can you share offline your cluster setup (topology) with CS configuration, so that i can test it further.

Well i understand that in the later patches we are targetting it more as partition than labels, but in that case shall i modify the same in other locations of WEB like node labels page, in CS page shall i mark it as Accessible Partitions ?

Good point, I think we may need keep it to be label, and do the renaming in a separated patch.

in CS page shall i mark it as Accessible Partitions

We can keep calling it "label" avoid confusion.

you mean if no node is mapped to cluster node label then not to show that Node Label ?

In my mind is, show all node labels no matter they mapped to nodes/queues or not. We can optimize this easily in the future, I prefer to keep completed message before people post their comments.

you mean the existing names of metrics entries needs to be appended with (Partition=xxx) and not to show both right ?

I think we need to show both (partition-specific and queue general), the only change is append with (Node-Label=xxx).

Its great to hear its working fine, but it worked without any modifications to the patch ?

Forgot to mention, I modified patch a little bit, removed some avoid-displaying-checking mentioned by you at https://issues.apache.org/jira/browse/YARN-3362?focusedCommentId=14517364&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14517364.
Uploading modified patch as well as CS config for you to test.

Hi Tan, Wangda,
Please find the attached patch for the labels (tested with 2 nodes cluster), and some pending work (may be can raise seperate jira to handle them as you suggested earlier)

1. I feel along with the labels we can specify how much resources are available, as now we need to go node labels page and come back to get to know it
2. Support showing show app-by-queue-by-label (Filter apps in a queue by label needs add additional CS interface,)
3. active-user-info by queue by label (active apps, pending apps & USER AM resource limit)

-1 overall

This message was automatically generated.

Can you please post the latest screenshot?

Hi Naga,
Thanks for updating, I took a look at closer look at latest patch, and tried in my local cluster, some comments in code:
1) There's some duplicated code for CapacitySchedulerInfo, you only need keep one constructor, which takes label always
2) Also, protected CapacitySchedulerQueueInfoList getQueues doesn't need to do specifial logic when null == nodeLabel, queue can be ignored when: queue cannot access to label AND label is exclusive-label. accessibleToPartition in AbstractCSQueue already considered when label==""
3) CapacitySchedulerQueueInfo doesn't need two constructor as well
4) When you want to get "NO_LABEL", you should pass "RMNodeLabelsManager.NO_LABEL" instead of null.

Some comments in UI:
1) I've reconsidered the hierarchy, I think it's better to rename "Node_Label=<...>" to "Partition:<...>". For the Node Labels page, we should add a column which is "Node Label Type", and indicate they're partition.
2) <NO_LABEL> should be <DEFAULT_PARTITION>, <NO_LABEL> itself may not clear enough, also in Node Labels page, we should change <NO_LABEL> to the same name here. Changes in Node Labels page could add in separated JIRA.
3) It's better to indicate queue in hierarchy also, now we have:

+ Partition: xxx
  + root
     + a
     + b

I think it's better to add a "Queue:" before queue name:

+ Partition: xxx
  + Queue: root
     + Queue: a
     + Queue: b

Otherwise people may get confused which one is partition and which one is queue.

4) In queue's metrics overview table, it mixing partition-specific and queue-generic metrics is not clear enough, I'm not sure if you can show it like this (If existing Hamlet framework supports this or not):

+ Partition: xxx
  + Queue: root
     + Queue: a
       |---------  Partition Specific Metrics -----------|
       |  Used Capacity      |    0.0%                   |
       |  Absolute Capacity  |   50.0%                   |
                   ........
       |-------------------------------------------------|

       |---------  Queue General Metrics ----------------|
       |          Other Metrics                          |
       ---------------------------------------------------

       Active user info:
       ........ Table ..........

     + Queue: b

Attached folded/expanded queues screenshots.

Hi Tan, Wangda,
Please find the new patch addressing your previous comments, Will raise new jira for handling modifications in the Node Labels Page.

queue can be ignored when: queue cannot access to label AND label is exclusive-label. accessibleToPartition in AbstractCSQueue already considered when label==""

May be i did not get this completely. label is exclusive-label i have done in CapacitySchedulerPage.QueuesBlock.render(l num - 357)

+1 overall

This message was automatically generated.

Hi Naganarasimha G R,
Thanks a lot for updating, looks much better now! I still have few minor comments:

For UI:

1) "Configured Capacity", "Configured Max Capacity" should be a part of "Queue Status for Partition..."?
2) Not caused by your patch, Absolute Capacity should be "Absolute Configured Capacity", Absolute Max Capacity should be "Absolute Configured Max Capacity", could you update them in your patch?
3) Also not caused by your patch, there's no space between "Active Users Info" and following queue, I'm not sure if there's any easy fix can do, please feel free to file a separate ticket if it will be hard to be solved together.

For implementation:
1) One minor comment for style is, you can merge all capacities-related rendering in CapacitySchedulerPage to a method similar to renderCommonLeafQueueInfo, which you can merge some implementation for render and renderLeafQueueInfoWithoutParition. And add a method renderLeafQueueInfoWithParition to make render looks cleaner.

For your question

May be i did not get this completely. label is exclusive-label i have done in CapacitySchedulerPage.QueuesBlock.render(l num - 357)

I think for both CapacitySchedulerPage and CapacitySchedulerInfo, it should be:

	if (label.getIsExclusive()
        && !((AbstractCSQueue) root).accessibleToPartition(label.getLabelName())) {

When label is exclusive (nobody can use the label except queue is accessible to the label) and (queue isn't accessible to the label), we don't need to continue.

Let me know your thoughts.

CC: Vinod Kumar Vavilapalli/Jian He.

Attached "https://issues.apache.org/jira/secure/attachment/12731236/No-space-between-Active_user_info-and-next-queues.png" to showing there's no space between Active User Info and next queue.

Thanks for the review Tan, Wangda,

Also not caused by your patch, there's no space between "Active Users Info" and following queue, I'm not sure if there's any easy fix can do, please feel free to file a separate ticket if it will be hard to be solved together.

Put some considerable amount of effort, dint workout. Hamlet should have add some kind of doc or a book, its seems like an enigma code to me. Will file a separate jira as its required in many places of CS page : "Active Users Info" and following queue, in between Dump scheduler log button and Application Queue , In CS Health Block
For other comments will rework ASAP.

Put some considerable amount of effort, dint workout. Hamlet should have add some kind of doc or a book, its seems like an enigma code to me. Will file a separate jira as its required in many places of CS page : "Active Users Info" and following queue, in between Dump scheduler log button and Application Queue , In CS Health Block

Thanks for looking at this, that sounds good, we can address of them together in a separated JIRA.

Hi Tan, Wangda
Updating the patch based on your review comments
While testing came across few issues

• why all labels are default accessible @ root? currently have removed label.getIsExclusive() && !((AbstractCSQueue) root).accessibleToPartition(label) check from CapacitySchedulerPage for root queue for each label as its always true
• currently accessibility of labels for non root queues, if not specified is * as it inherits from parent queue. Is this right ?
• when new partition is added, then the configured max capacity is shown as 100 but abs max capacity is shown as 0 dint understand why this is handled in this way in UI side CapacitySchedulerQueueInfo(ln 73)

    
    if (maxCapacity < EPSILON || maxCapacity > 1f)
        maxCapacity = 1f;
  

  my guess is code has been added to show that its in the range of 0 to 1f ? if so maxCapacity < EPSILON then maxCapacity=0 and if maxCapacity > 1f then maxCapacity=1f right ?
  Also my doubt is why keep max capacity as zero if label is accessible to a queue, if max capacity is not specified then it should be 100 right ?

Also few suggestions in UI :

• Earlier over all available resource info was available in cluster metrics and hence i think it was easier for viewer to relate the percentages but here available resource info for a given label is present only in nodelabels page. so i would suggest to have label resource info like :

  + Partition: xxx [memory:8192, vCores:8]
    + Queue: root
       + Queue: a
       + Queue: b
  + Partition: yyy [memory:4096, vCores:8]
    + Queue: root
       + Queue: a
       + Queue: b
  

• I think queue hierarchy is not always helpful when i want to relatively compare capacities among leaf queue (through bars), so i would suggest additional view without queue hierarchy when user clicks on a button [ kind of toggle button].

                  -------------------
                  | Hide Hierarchy |
                  -------------------
  + Partition: xxx [memory:8192, vCores:8]
    + Queue Path: root.Q1.a
      ....
    + Queue Path: default
  + Partition: yyy [memory:4096, vCores:8]
    + Queue Path: root.Q1.a
      ....
    - Queue Path: default
         |---------  Partition Specific Metrics -----------|
         |                 ........                        |
         |-------------------------------------------------|
         |---------  Queue General Metrics ----------------|
         |                 ........                        |
         |-------------------------------------------------|
         Active user info:
         ........ Table ..........
  

-1 overall

This message was automatically generated.

Fixing the valid check style issues for the patch (more than 80 chars are not corrected for readability purposes)

-1 overall

This message was automatically generated.

Hi Naga,
Thanks for updating,

1) To your questions: https://issues.apache.org/jira/browse/YARN-3362?focusedCommentId=14537181&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14537181,
You can refer to YARN-2824 for more information about why default cap of labeled resource set to zero.
The default of max-cap is 100 because queue can use such resource without configure it. Let me know if you have more questions.

2) About showing resources of partitions, I think it's very helpful. I think you can include used-resource of each partition as well, You can file a separate ticket if it is hard to be added with this ticket.

3) About "Hide Hierarchy", I think it's good for queue capacity comparison, but admin may get confused after checked "Hide Hierarchy", it's better to be added to some other places instead of modify queue UI itself.

The latest patch LGTM.

Thanks for the comment Tan, Wangda, Here are my comments inline

The default of max-cap is 100 because queue can use such resource without configure it. Let me know if you have more questions.

IIUC, the current code only shows in the UI as 100% that too at the location which i mentioned ( CapacitySchedulerQueueInfo ln 73) but actual absolute capacity is zero which makes like no bar is shown. I think its a bug in the UI and as well it should be in the way we are discussing(max-cap is 100 because queue can use such resource without configure it), let me test and raise it.

About showing resources of partitions, I think it's very helpful. I think you can include used-resource of each partition as well, You can file a separate ticket if it is hard to be added with this ticket.

I think its a simple change, give me some time will get the modifications done.

About "Hide Hierarchy", I think it's good for queue capacity comparison, but admin may get confused after checked "Hide Hierarchy", it's better to be added to some other places instead of modify queue UI itself.

I meant, label on the button should also be toggling, i.e. when hierarchies are shown then label can be Hide Hierarchy / Show only Leaf Queues and when only Leaf Queues we can have label as Show Queue Hierarchy. Well i think if the idea is ok can raise and handle in another jira as it involves lot more discussions but i felt its particularly useful in our labels view as there are too many things displayed in the UI and more the labels or more queue hierarchy then it will be come more chaotic.

Hi Tan, Wangda
Have updated the patch and image with appending label with its available resource

-1 overall

This message was automatically generated.

Naganarasimha G R, thanks for updating, the latest result looks great!

About the queue-hierarchy discussion, I think maybe one alternative is, make a hierarchy queue names, but align usage bars, like following:

root        [------------------------------ 100% used]
  - a       [----------------- 60% used]
    - a1    [------------- 40% used]
    - a2    [-----------]
  - b       [-------------]
    - b1    [---------]
      - b11 [----------]

Which can also help comparing queue's resource but doesn't need a extra button to hide/show queue hierarchy?

Thanks for the comment Tan, Wangda,
This idea is also good but i can think of few issues here

• In most cases Queue hierarchy will be collapsed so the space between non leaf queue name and the bar might not look good
• Bigger the name of the leaf queues more space between the queue name and the bar (at the upper level queues)
• Space based calculation might not work out so implementing it with Hammock might be little complex.
• Here were you thinking of publishing absolute usage or usage @ queue level, I feel absolute usage would be better (YARN-3638 also suggest it), but existing shows just usage @ queue level.

Also i feel these modifications can take time, if this jira is important then we can push this in and we can further discuss about UI enhancements in another jira. thoughts?

Naganarasimha G R,
Thanks for pointing YARN-3638, I think what you want can be addressed or at least more related YARN-3638, right? I suggest to move discussion to YARN-3638 to make this JIRA more concentrated.

Last call for comments, as I'm planning to commit today.

Committed to trunk/branch-2, thanks Naganarasimha G R and review from Vinod Kumar Vavilapalli.

FAILURE: Integrated in Hadoop-trunk-Commit #7825 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7825/)
YARN-3362. Add node label usage in RM CapacityScheduler web UI. (Naganarasimha G R via wangda) (wangda: rev 0e85044e26da698c45185585310ae0e99448cd80)

• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
• hadoop-yarn-project/CHANGES.txt
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java

Thanks for reviewing and committing the jira, Tan, Wangda, And Yes it would be better to discuss these CS queue hierarchy modifications in YARN-3638, will update our discussions there.

FAILURE: Integrated in Hadoop-Yarn-trunk #927 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/927/)
YARN-3362. Add node label usage in RM CapacityScheduler web UI. (Naganarasimha G R via wangda) (wangda: rev 0e85044e26da698c45185585310ae0e99448cd80)

• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
• hadoop-yarn-project/CHANGES.txt
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java

SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #196 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/196/)
YARN-3362. Add node label usage in RM CapacityScheduler web UI. (Naganarasimha G R via wangda) (wangda: rev 0e85044e26da698c45185585310ae0e99448cd80)

• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
• hadoop-yarn-project/CHANGES.txt
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #195 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/195/)
YARN-3362. Add node label usage in RM CapacityScheduler web UI. (Naganarasimha G R via wangda) (wangda: rev 0e85044e26da698c45185585310ae0e99448cd80)

• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
• hadoop-yarn-project/CHANGES.txt
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java

FAILURE: Integrated in Hadoop-Hdfs-trunk #2125 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2125/)
YARN-3362. Add node label usage in RM CapacityScheduler web UI. (Naganarasimha G R via wangda) (wangda: rev 0e85044e26da698c45185585310ae0e99448cd80)

• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
• hadoop-yarn-project/CHANGES.txt
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #185 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/185/)
YARN-3362. Add node label usage in RM CapacityScheduler web UI. (Naganarasimha G R via wangda) (wangda: rev 0e85044e26da698c45185585310ae0e99448cd80)

• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
• hadoop-yarn-project/CHANGES.txt
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java

SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2143 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2143/)
YARN-3362. Add node label usage in RM CapacityScheduler web UI. (Naganarasimha G R via wangda) (wangda: rev 0e85044e26da698c45185585310ae0e99448cd80)

• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
• hadoop-yarn-project/CHANGES.txt
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java

Wangda Tan and Sunil G, as we discussed in YARN-4751, I am attaching the 2.7 backport of YARN-3362 to this JIRA. Please see YARN-4751 to for a discussion of the additional changes needed for 2.7 that will provide accurate metrics on labeled queues.

2.7-specific differences in YARN-3362-branch-2.7.002.patch:

• The RMNodeLabel class is named NodeLabel in 2.7.
• The CapacitySchedulerInfo constructor doesn't have the CapacityScheduler parameter in 2.7, so I left it out.
• One of the changes for CapacitySchedulerInfo#getQueues in YARN-3362 added a check to skip non-accessible queues. In 2.7, there are 2 loops that iterate over the queues whereas in 2.8 and beyond, there is only 1. I put the check in the first loop because the first loop is filtering the queues to be processed by the second loop

+ Naganarasimha G R.

Thanks Eric Payne for sharing patch here. Very much appreciate the same.
Overall patch looks fine for me. Will wait for NGarla_Unused also.

Thanks Eric Payne, Tan, Wangda, & Sunil G for waiting, taking a look at it shortly ...

Hi Eric Payne, i was trying to compile, install and test, one major blocker was :

• When i submit a app to a queue with default node label expression set to a partition then in the scheduler page stats was not getting displayed.

Other issues which are good to be backported :

1. i think there is one other jira which was capturing the queue's default node label expression that would also be helpful to be back ported
2. App's AM's label is also good to be captured in the web ui and displayed.
3. would it be good to back port MAPREDUCE-6304 ?

attached AppInLabelXnoStatsInSchedPage.png for the same tried submitting to default queue and other hierarchical queue both had the same behavior

• When i submit a app to a queue with default node label expression set to a partition then in the scheduler page stats was not getting displayed.

Other issues which are good to be backported :

1. i think there is one other jira which was capturing the queue's default node label expression that would also be helpful to be back ported
2. App's AM's label is also good to be captured in the web ui and displayed.
3. would it be good to back port MAPREDUCE-6304 ?

Naganarasimha G R, thank you very much for your review and comments.

I'm sorry I did not make it clear in this JIRA. Please refer to conversations in YARN-4751 between me, Wangda Tan, and Sunil G.

The problem is that in order to provide all of the needed functionality in 2.7, we either have to backport several JIRAs which would destabilize 2.7 (please see this comment and this comment) or we need to pick and choose pieces to backport without doing full cherry-picks.

After the discussion in YARN-4751, we thought it would be best to submit a 2.7 patch here that only backported YARN-3362 and then add the fixes for the metrics in YARN-4751. Please let me know what you think.

Hi Eric Payne,
Thanks for providing the back ground, i went through the comments which you mentioned but still not able to understand whether the patch what you have shared would work independently ? After applying the patch then partition usage information needs to be displayed for the queue under the partition. But after applying the patch i was not able to see the stats in the webui even though the application is submitted to partition and it is running refer AppInLabelXnoStatsInSchedPage.png. And if i submit to default parition then webui stats are proper.
If another patch is required to get the webui working, may be you can share the patch then i can review. Also dint have time to cross check why it was not working if required can do the analysis.

And for other jira's its good to have and not compulsary

After applying the patch then partition usage information needs to be displayed for the queue under the partition. But after applying the patch i was not able to see the stats in the webui even though the application is submitted to partition and it is running

Naganarasimha G R, in order to see the stats and bar graph metrics, both YARN-3362-branch-2.7.002.patch and YARN-4751-branch-2.7.004.patch need to be applied. First YARN-3362-branch-2.7.002.patch should be applied and then YARN-4751-branch-2.7.004.patch. Did you do that and are still seeing problems?

If you think it would be cleaner, I can combine the patches and submit it as one patch here in YARN-3362.

Oops let me apply both and test i had applied only the current patch(YARN-3362-branch-2.7.002.patch ). Thanks for the update!

I tested with both patches, and metrics was coming fine (tested w/o labels and with labels). But when I uploaded only YARN-3362 patch, then same pblm came as NGarla_Unused has mentioned.

In YARN-4751, AbstractCSQueue changes helps to display queue Used Capacity correctly. I think that was the differentiated change for this.

If you think it would be cleaner, I can combine the patches and submit it as one patch here in YARN-3362.

I think its looks fine for me.

If you think it would be cleaner, I can combine the patches and submit it as one patch here in YARN-3362.

I think its looks fine for me.

Thanks, Sunil G, for your comments. Naganarasimha G R, given that we will keep the patches separate, do you want to comment further on the 2.7 patch for YARN-3362?

Will have chk once, please wait till EOD...

Naganarasimha Garla, is there any update?

Sorry for the long delay, I verified the patch in the order you have mentioned, i built and crossverified the patch on 2.7 and i am fine with the modifications too.
Just one query is how to go about with the merge? would it be good to reopen this jira and change the state to patch available to so that the jenkins is triggered or merge these changes with YARN-4751. I would prefer the later , thoughts ?
cc / Tan, Wangda

Naganarasimha G R,

I would prefer the previous one: reopen this ticket, kick Jenkins and change status of this JIRA. YARN-4751 can be committed separately. Typically we should backport JIRA to older version with same ID, changing ID sometimes causes trouble when doing back port.

Thoughts?

fine with me.. reopening the ticket and starting the jenkins!

To backport it to 2.7 branch

manually started jenkins build as it had not yet started.

Thanks a lot, Naganarasimha G R.

I don't think the YARN pre-commit build worked. I the following error message in the console output from both of the following builds:
https://builds.apache.org/job/PreCommit-YARN-Build/11301/console
https://builds.apache.org/job/PreCommit-YARN-Build/11304/console

Thanks Eric Payne, Reattaching the same file so that jenkins triggers with proper attachment.

-1 overall

This message was automatically generated.

Hi Eric Payne, Can you have a look at checkstyle, may be only few are rightly related to the patch ?

Naganarasimha G R], uploading YARN-3362-branch-2.7.003.patch

Most of the checkstyle warnings were from the previous (trunk) patch. However, there were a few that I introduced, so I fixed those. I also fixed some of the other ones including methods and parameters that needed to be final and adding minimal javadocs.

-1 overall

This message was automatically generated.

Naganarasimha G R, attaching YARN-3362-branch-2.7.004.patch with another checkstyle change correcting the order of final and protected.

Once pre-commit build comes back, can you please review?

Sure will review as it's also close can commit it too...

-1 overall

This message was automatically generated.

Thanks, Naganarasimha Garla. Have you been able to look at this latest patch?

Hi Eric Payne, As in earlier patches findbugs and test case are not related to the patch, latest patch LGTM. committing it shortly !

Have committed the patch to 2.7 branch, Thanks Eric Payne, for working on 2.7 patch. Thanks Sunil G and Tan, Wangda for the review.
Also we need to push YARN-4751 asap, as current patch is not complete without it

will close this jira once YARN-4751 is committed and closed...

YARN-4751 has been checked in hence now label UI will be available in 2.7 too

Changing Fix Version to 2.7.3 since branch 2.7.3 has not yet been created.

Closing the JIRA as part of 2.7.3 release.