Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5788

Apps not activiated and AM limit resource in UI and REST not updated after -replaceLabelsOnNode

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None

      Description

      Steps to reproduce
      ==============

      1. Enable node labels
      2. Configure capacity scheduler xml with label capacity configuration
      3. Add labelx to cluster
      4. Replace Node labels on node1
      5. Check scheduler and Rest API

      Actual
      Am limit in scheduler UI is still based on old resource
      Expected
      AM limit to be updated based new partition resource

      1. Screenshot from 2016-10-27 22-31-46.png
        166 kB
        Bibin A Chundatt
      2. YARN-5788.0001.patch
        9 kB
        Bibin A Chundatt
      3. YARN-5788.0002.patch
        6 kB
        Bibin A Chundatt
      4. YARN-5788.0003.patch
        10 kB
        Varun Saxena

        Issue Links

          Activity

          Hide
          varun_saxena Varun Saxena added a comment - - edited

          Thanks Bibin A Chundatt for raising the issue.
          I guess this is because we calculate AM limits only while calling activating applications.
          So if no app comes, AM limit wont be updated.

          Show
          varun_saxena Varun Saxena added a comment - - edited Thanks Bibin A Chundatt for raising the issue. I guess this is because we calculate AM limits only while calling activating applications. So if no app comes, AM limit wont be updated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Yes we have to update the same on node labels update also.root.clusterResource should be able handle.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Yes we have to update the same on node labels update also.root.clusterResource should be able handle.
          Hide
          varun_saxena Varun Saxena added a comment - - edited

          Agree in principle, Bibin A Chundatt. Replacing labels on node will lead to change in partition resources and this should be treated the same as addition or removal of a node which leads to change in overall cluster resources.

          Show
          varun_saxena Varun Saxena added a comment - - edited Agree in principle, Bibin A Chundatt . Replacing labels on node will lead to change in partition resources and this should be treated the same as addition or removal of a node which leads to change in overall cluster resources.
          Hide
          bibinchundatt Bibin A Chundatt added a comment - - edited

          Attaching patch after following update

          1. Added new interface in csqueue to refreshResources
          2. Currently only AM Limits update is added.
          3. Activate application when node labels are modified for nodemanager
          Show
          bibinchundatt Bibin A Chundatt added a comment - - edited Attaching patch after following update Added new interface in csqueue to refreshResources Currently only AM Limits update is added. Activate application when node labels are modified for nodemanager
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 46s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 36s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 0m 56s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 29s the patch passed
          +1 javac 0m 29s the patch passed
          -0 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 234 unchanged - 1 fixed = 237 total (was 235)
          +1 mvnsite 0m 35s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 3s the patch passed
          +1 javadoc 0m 19s the patch passed
          -1 unit 24m 57s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          40m 18s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
            org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
            org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimitsByPartition
            org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
            org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel
            org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
            org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities
            org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA
            org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
            org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue YARN-5788
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835971/YARN-5788.0001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 76bd495b6c98 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / aa3cab1
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13664/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13664/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13664/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13664/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 46s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 56s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 29s the patch passed +1 javac 0m 29s the patch passed -0 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 234 unchanged - 1 fixed = 237 total (was 235) +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 3s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 24m 57s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 40m 18s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation   org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimitsByPartition   org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation   org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel   org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService   org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities   org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA   org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate   org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5788 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835971/YARN-5788.0001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 76bd495b6c98 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aa3cab1 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13664/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13664/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13664/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13664/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching patch after correcting testcase failur

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching patch after correcting testcase failur
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Uploading patch after changing patch name

          Show
          bibinchundatt Bibin A Chundatt added a comment - Uploading patch after changing patch name
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 44s trunk passed
          +1 compile 0m 31s trunk passed
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 0m 37s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 0m 54s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 29s the patch passed
          +1 javac 0m 29s the patch passed
          +1 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 101 unchanged - 1 fixed = 101 total (was 102)
          +1 mvnsite 0m 34s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 3s the patch passed
          +1 javadoc 0m 17s the patch passed
          +1 unit 38m 19s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          53m 17s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue YARN-5788
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835978/YARN-5778.0002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0650e7af44ce 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / aa3cab1
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13668/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13668/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 44s trunk passed +1 compile 0m 31s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 37s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 54s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 29s the patch passed +1 javac 0m 29s the patch passed +1 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 101 unchanged - 1 fixed = 101 total (was 102) +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 3s the patch passed +1 javadoc 0m 17s the patch passed +1 unit 38m 19s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 53m 17s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5788 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835978/YARN-5778.0002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0650e7af44ce 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aa3cab1 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13668/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13668/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 43s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 38s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 0m 57s trunk passed
          +1 javadoc 0m 20s trunk passed
          +1 mvninstall 0m 32s the patch passed
          +1 compile 0m 30s the patch passed
          +1 javac 0m 30s the patch passed
          +1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 101 unchanged - 1 fixed = 101 total (was 102)
          +1 mvnsite 0m 35s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 4s the patch passed
          +1 javadoc 0m 18s the patch passed
          -1 unit 35m 2s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          50m 15s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue YARN-5788
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835983/YARN-5788.0002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1267279af1de 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / aa3cab1
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13669/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13669/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13669/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 43s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 57s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed +1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 101 unchanged - 1 fixed = 101 total (was 102) +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 4s the patch passed +1 javadoc 0m 18s the patch passed -1 unit 35m 2s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 50m 15s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5788 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835983/YARN-5788.0002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1267279af1de 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aa3cab1 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/13669/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13669/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13669/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment - - edited

          Thanks Bibin A Chundatt for the patch.
          Although updateClusterResource would handle just about everything and easiest way to handle this but as cluster resource will remain the same on replace labels on node, do we need to carry out each step in updateClusterResource ?
          For instance, queue usage related ratios and capacities for instance need to be updated only for 2 sets of relevant partitions for each node i.e. the old and new/replaced one. This can be useful to bypass queues where both relevant partitions are not applicable.
          We should activate apps as this some apps can be activated with replace label on node. This is a heavy operation and we will hold a write lock on scheduler for this but this should be fine as replace labels on node should not be a frequent operation.

          cc Wangda Tan. Your thoughts on this ?

          Show
          varun_saxena Varun Saxena added a comment - - edited Thanks Bibin A Chundatt for the patch. Although updateClusterResource would handle just about everything and easiest way to handle this but as cluster resource will remain the same on replace labels on node, do we need to carry out each step in updateClusterResource ? For instance, queue usage related ratios and capacities for instance need to be updated only for 2 sets of relevant partitions for each node i.e. the old and new/replaced one. This can be useful to bypass queues where both relevant partitions are not applicable. We should activate apps as this some apps can be activated with replace label on node. This is a heavy operation and we will hold a write lock on scheduler for this but this should be fine as replace labels on node should not be a frequent operation. cc Wangda Tan . Your thoughts on this ?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for raising and working on it Bibin A Chundatt,
          Agree with Varun Saxena, though it affects performance but IMO impact is less as its called once for the cli command run (though multiple node mapping can be there). Also we can note that the same happens for NodeAdded event too (where in we could have called for specific partition alone).
          So i am ok with going ahead with this fix here and if it really impacting performance taking it of in seperate jira!

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for raising and working on it Bibin A Chundatt , Agree with Varun Saxena , though it affects performance but IMO impact is less as its called once for the cli command run (though multiple node mapping can be there). Also we can note that the same happens for NodeAdded event too (where in we could have called for specific partition alone). So i am ok with going ahead with this fix here and if it really impacting performance taking it of in seperate jira!
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Bibin A Chundatt for reporting/working on the patch and Varun Saxena/Naganarasimha G R for reviews.

          The latest patch looks good to me, +1.

          I agree that the node label update is not a very frequency op in known use cases. And this patch only refreshes cluster resource once per batch update.

          Thoughts?

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Bibin A Chundatt for reporting/working on the patch and Varun Saxena / Naganarasimha G R for reviews. The latest patch looks good to me, +1. I agree that the node label update is not a very frequency op in known use cases. And this patch only refreshes cluster resource once per batch update. Thoughts?
          Hide
          varun_saxena Varun Saxena added a comment -

          Thanks Naganarasimha G R and Wangda Tan for sharing your views.

          Even I agree performance wise it should be fine because calls for replace labels on node should be infrequent and in a batch.

          note that the same happens for NodeAdded event too (where in we could have called for specific partition alone).

          True that.

          Will commit it later today unless there are further comments.

          Show
          varun_saxena Varun Saxena added a comment - Thanks Naganarasimha G R and Wangda Tan for sharing your views. Even I agree performance wise it should be fine because calls for replace labels on node should be infrequent and in a batch. note that the same happens for NodeAdded event too (where in we could have called for specific partition alone). True that. Will commit it later today unless there are further comments.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 53s trunk passed
          +1 compile 0m 36s trunk passed
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 41s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 0m 59s trunk passed
          +1 javadoc 0m 20s trunk passed
          +1 mvninstall 0m 32s the patch passed
          +1 compile 0m 30s the patch passed
          +1 javac 0m 30s the patch passed
          +1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 101 unchanged - 1 fixed = 101 total (was 102)
          +1 mvnsite 0m 35s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 1s The patch has no whitespace issues.
          +1 findbugs 1m 2s the patch passed
          +1 javadoc 0m 18s the patch passed
          +1 unit 36m 3s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          51m 35s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue YARN-5788
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836311/YARN-5788.0003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0f0e2ef30736 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7ba74be
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13722/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13722/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 53s trunk passed +1 compile 0m 36s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 41s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 0m 59s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed +1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 101 unchanged - 1 fixed = 101 total (was 102) +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 findbugs 1m 2s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 36m 3s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 51m 35s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5788 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836311/YARN-5788.0003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0f0e2ef30736 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7ba74be Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13722/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13722/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 51s trunk passed
          +1 compile 0m 33s trunk passed
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 39s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 1m 1s trunk passed
          +1 javadoc 0m 20s trunk passed
          +1 mvninstall 0m 31s the patch passed
          +1 compile 0m 30s the patch passed
          +1 javac 0m 30s the patch passed
          +1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 101 unchanged - 1 fixed = 101 total (was 102)
          +1 mvnsite 0m 40s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 20s the patch passed
          +1 javadoc 0m 22s the patch passed
          +1 unit 36m 26s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          53m 16s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue YARN-5788
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836311/YARN-5788.0003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0c6fe5fa071a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7ba74be
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13723/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13723/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 51s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 39s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 1s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 31s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed +1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 101 unchanged - 1 fixed = 101 total (was 102) +1 mvnsite 0m 40s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 20s the patch passed +1 javadoc 0m 22s the patch passed +1 unit 36m 26s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 53m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5788 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836311/YARN-5788.0003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0c6fe5fa071a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7ba74be Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13723/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13723/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          HI Bibin A Chundatt
          In latest patch, I could see that lock is removed for updateLabelsOnNode. Is this intentional?

          Show
          sunilg Sunil G added a comment - HI Bibin A Chundatt In latest patch, I could see that lock is removed for updateLabelsOnNode . Is this intentional?
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10741 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10741/)
          YARN-5788. Apps not activiated and AM limit resource in UI and REST not (varunsaxena: rev 7d2d8d25ba0cb10a3c6192d4123f27ede5ef2ba6)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNodeLabelUpdate.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10741 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10741/ ) YARN-5788 . Apps not activiated and AM limit resource in UI and REST not (varunsaxena: rev 7d2d8d25ba0cb10a3c6192d4123f27ede5ef2ba6) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNodeLabelUpdate.java
          Hide
          varun_saxena Varun Saxena added a comment -

          Sunil G, updateLabelsOnNode is only called from updateNodeLabelsAndQueueResource which is protected by write lock. So this should be fine.

          Show
          varun_saxena Varun Saxena added a comment - Sunil G , updateLabelsOnNode is only called from updateNodeLabelsAndQueueResource which is protected by write lock. So this should be fine.
          Hide
          varun_saxena Varun Saxena added a comment -

          Committed to trunk, branch-2.
          Thanks Bibin A Chundatt for your contribution. Thanks Sunil G, Wangda Tan and Naganarasimha Garla for reviews.

          Show
          varun_saxena Varun Saxena added a comment - Committed to trunk, branch-2. Thanks Bibin A Chundatt for your contribution. Thanks Sunil G , Wangda Tan and Naganarasimha Garla for reviews.

            People

            • Assignee:
              bibinchundatt Bibin A Chundatt
              Reporter:
              bibinchundatt Bibin A Chundatt
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development