Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4624

NPE in PartitionQueueCapacitiesInfo while accessing Schduler UI

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Scenario:
      =======

      Configure nodelables and add to cluster
      Start the cluster

      java.lang.NullPointerException
      	at org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.PartitionQueueCapacitiesInfo.getMaxAMLimitPercentage(PartitionQueueCapacitiesInfo.java:114)
      	at org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderQueueCapacityInfo(CapacitySchedulerPage.java:163)
      	at org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderLeafQueueInfoWithPartition(CapacitySchedulerPage.java:105)
      	at org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.render(CapacitySchedulerPage.java:94)
      	at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
      	at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
      	at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
      	at org.apache.hadoop.yarn.webapp.view.HtmlBlock$Block.subView(HtmlBlock.java:43)
      	at org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
      	at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$LI._(Hamlet.java:7702)
      	at org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$QueueBlock.render(CapacitySchedulerPage.java:293)
      	at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
      	at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
      	at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
      	at org.apache.hadoop.yarn.webapp.view.HtmlBlock$Block.subView(HtmlBlock.java:43)
      	at org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
      	at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$LI._(Hamlet.java:7702)
      	at org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$QueuesBlock.render(CapacitySchedulerPage.java:447)
      	at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
      	at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
      	at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
      
      1. YARN-4624-003.patch
        3 kB
        Brahma Reddy Battula
      2. YARN-4624.patch
        3 kB
        Brahma Reddy Battula
      3. YARN-4624.4.patch
        3 kB
        Sunil G
      4. YARN-2674-002.patch
        2 kB
        Brahma Reddy Battula
      5. SchedulerUIWithOutLabelMapping.png
        103 kB
        Sunil G

        Issue Links

          Activity

          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Naganarasimha Garla thanks a lot for review and commit.. and thanks to Sunil G and others.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Naganarasimha Garla thanks a lot for review and commit.. and thanks to Sunil G and others.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the contributions Brahma & Sunil G and review from Sunil G ,Devaraj K, Rohith Sharma K S & Bibin A Chundatt.
          Commited to 2.8, branch-2 & trunk

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the contributions Brahma & Sunil G and review from Sunil G , Devaraj K , Rohith Sharma K S & Bibin A Chundatt . Commited to 2.8, branch-2 & trunk
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10227 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10227/)
          YARN-4624. NPE in PartitionQueueCapacitiesInfo while accessing Schduler (naganarasimha_gr: rev d81b8163b4e5c0466a6af6e1068f512c5fd24a61)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionQueueCapacitiesInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueCapacitiesInfo.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10227 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10227/ ) YARN-4624 . NPE in PartitionQueueCapacitiesInfo while accessing Schduler (naganarasimha_gr: rev d81b8163b4e5c0466a6af6e1068f512c5fd24a61) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionQueueCapacitiesInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueCapacitiesInfo.java
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the patch Sunil G,
          Applied patch compiled and tested in an installation, seems like everything is working on top of trunk code.
          If no one has any concerns will commit this patch tomorrow !

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the patch Sunil G , Applied patch compiled and tested in an installation, seems like everything is working on top of trunk code. If no one has any concerns will commit this patch tomorrow !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 51s trunk passed
          +1 compile 0m 37s trunk passed
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 44s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 1m 3s trunk passed
          +1 javadoc 0m 23s trunk passed
          +1 mvninstall 0m 31s the patch passed
          +1 compile 0m 33s the patch passed
          +1 javac 0m 33s the patch passed
          +1 checkstyle 0m 20s the patch passed
          +1 mvnsite 0m 42s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 4s the patch passed
          +1 javadoc 0m 19s the patch passed
          +1 unit 37m 28s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          53m 38s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821778/YARN-4624.4.patch
          JIRA Issue YARN-4624
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3d20820f93a0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4ef1bc0
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12623/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12623/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 51s trunk passed +1 compile 0m 37s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 44s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 3s trunk passed +1 javadoc 0m 23s trunk passed +1 mvninstall 0m 31s the patch passed +1 compile 0m 33s the patch passed +1 javac 0m 33s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 42s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 4s the patch passed +1 javadoc 0m 19s the patch passed +1 unit 37m 28s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 53m 38s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821778/YARN-4624.4.patch JIRA Issue YARN-4624 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3d20820f93a0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4ef1bc0 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12623/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12623/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment - - edited

          Yes NGarla_Unused. Thanks for the update. Attaching a rebased patch given by Brahma Reddy Battula. Test case is not needed as we are changes data type from boxed to normal float.

          Show
          sunilg Sunil G added a comment - - edited Yes NGarla_Unused . Thanks for the update. Attaching a rebased patch given by Brahma Reddy Battula . Test case is not needed as we are changes data type from boxed to normal float.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Sunil G, As discussed offline safer option is to go with patch 1, so can you rebase the patch so that we can make progress in this jira ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Sunil G , As discussed offline safer option is to go with patch 1, so can you rebase the patch so that we can make progress in this jira ?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          +1 for 2.8,

          Show
          Naganarasimha Naganarasimha G R added a comment - +1 for 2.8,
          Hide
          sunilg Sunil G added a comment -

          NGarla_Unused. As you know, with YARN-4304 there was an improvement planned to hide maxAMResourcePercenatageLimit for parent queue. In DAO object, if we can set the variable to null, we can achieve the same. Hence we chose Float and was setting null in case of ParentQueue. With v3 patch, we also wanted to do a null check in UI to avoid the pblm mentioned in this JIRA. So couple of typecasting needed there to convert from float to Float and findbugs reported this as error. we could hide this error to have this optimization in and I was mentioning for same in earlier comments. Only concern was that, we were doing some double type casting to achieve what we are expecting. So i thought, is its really good to do something like that which is not a clean code. Hence though of going with v1 patch. Am fine either way.
          Its good if we could also take input from Wangda Tan Brahma Reddy Battula and Bibin A Chundatt too who were involved and can come to conclusion for 2.8.

          I would like to keep target version as 2.8 if its fine in either way.

          Show
          sunilg Sunil G added a comment - NGarla_Unused . As you know, with YARN-4304 there was an improvement planned to hide maxAMResourcePercenatageLimit for parent queue. In DAO object, if we can set the variable to null, we can achieve the same. Hence we chose Float and was setting null in case of ParentQueue. With v3 patch, we also wanted to do a null check in UI to avoid the pblm mentioned in this JIRA. So couple of typecasting needed there to convert from float to Float and findbugs reported this as error. we could hide this error to have this optimization in and I was mentioning for same in earlier comments. Only concern was that, we were doing some double type casting to achieve what we are expecting. So i thought, is its really good to do something like that which is not a clean code. Hence though of going with v1 patch. Am fine either way. Its good if we could also take input from Wangda Tan Brahma Reddy Battula and Bibin A Chundatt too who were involved and can come to conclusion for 2.8. I would like to keep target version as 2.8 if its fine in either way.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Sunil G,

          we can group this new metrics (maxAMPercentageLimit etc) in other DAO object and can use it.

          If this is the alternate approach would it bring incompatibility later ? If so better to address in this jira and get it fixed as part of 2.8.
          Well if the approach was already discussed then we can go ahead with V3 patch and just add it to findbugs exclude file.

          Show
          Naganarasimha Naganarasimha G R added a comment - Sunil G , we can group this new metrics (maxAMPercentageLimit etc) in other DAO object and can use it. If this is the alternate approach would it bring incompatibility later ? If so better to address in this jira and get it fixed as part of 2.8. Well if the approach was already discussed then we can go ahead with V3 patch and just add it to findbugs exclude file.
          Hide
          sunilg Sunil G added a comment -

          Yes NGarla_Unused. We need to make some progress here, as its a long pending task.

          v3 patch comes with a find bugs issue. We do type conversion from to box type two times and causing this problem. I think we can go with v1 patch itself to resolve the issue for now. Improvement to hide some non-used items can be done in an another improvement jira. Thoughts?.
          Also patch might need a rebase, Brahma Reddy Battula, could you please help to rebase once v1 or v3 is decided. If you are busy, i could also help to do the same.

          Show
          sunilg Sunil G added a comment - Yes NGarla_Unused . We need to make some progress here, as its a long pending task. v3 patch comes with a find bugs issue. We do type conversion from to box type two times and causing this problem. I think we can go with v1 patch itself to resolve the issue for now. Improvement to hide some non-used items can be done in an another improvement jira. Thoughts?. Also patch might need a rebase, Brahma Reddy Battula , could you please help to rebase once v1 or v3 is decided. If you are busy, i could also help to do the same.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Sunil G & Tan, Wangda ,
          I feel NPE issue needs to be fixed before 2.8 and for other improvements(/fixes) we can tackle in other jira.
          I am fine with any approach which is taken as per v1 patch or v3 patch, May be Tan, Wangda can share his opinion so that we can finalize on the approach and commit it.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Sunil G & Tan, Wangda , I feel NPE issue needs to be fixed before 2.8 and for other improvements(/fixes) we can tackle in other jira. I am fine with any approach which is taken as per v1 patch or v3 patch, May be Tan, Wangda can share his opinion so that we can finalize on the approach and commit it.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 29s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 19s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 0m 35s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 58s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 31s the patch passed
          +1 compile 0m 29s the patch passed
          +1 javac 0m 29s the patch passed
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 0m 31s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 findbugs 1m 2s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 17s the patch passed
          -1 unit 34m 59s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 14s The patch does not generate ASF License warnings.
          49m 1s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderQueueCapacityInfo(ResponseInfo, String) At CapacitySchedulerPage.java:then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderQueueCapacityInfo(ResponseInfo, String) At CapacitySchedulerPage.java:[line 165]
          Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790103/YARN-4624-003.patch
          JIRA Issue YARN-4624
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ca825ce2fe6a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9581fb7
          Default Java 1.8.0_91
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11969/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11969/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11969/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11969/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11969/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11969/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 29s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 19s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 58s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 31s the patch passed +1 compile 0m 29s the patch passed +1 javac 0m 29s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 11s the patch passed -1 whitespace 0m 0s The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 findbugs 1m 2s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 17s the patch passed -1 unit 34m 59s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 49m 1s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderQueueCapacityInfo(ResponseInfo, String) At CapacitySchedulerPage.java:then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderQueueCapacityInfo(ResponseInfo, String) At CapacitySchedulerPage.java: [line 165] Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790103/YARN-4624-003.patch JIRA Issue YARN-4624 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ca825ce2fe6a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9581fb7 Default Java 1.8.0_91 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11969/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11969/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/11969/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11969/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11969/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11969/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          NGarla_Unused/Sunil G
          Came across the same issue while testing. Better to fix this jira before 2.8.
          Could you check the same.

          Show
          bibinchundatt Bibin A Chundatt added a comment - NGarla_Unused / Sunil G Came across the same issue while testing. Better to fix this jira before 2.8. Could you check the same.
          Hide
          sunilg Sunil G added a comment -

          I think this has gone stale. Lets move forward.

          In YARN-4304, one of the small improvement was to hide maxAMPercentageLimit for parent queue. We achieved this by declaring this variable as Float and was setting to null if queue is ParentQueue.

          Now we have got this corner scenario when labels are available in cluster, and no label-mappings are defined in CS. We have fixed partially in YARN-4634, however we can still have some more cases like here. So ideally its better to handle null check for getMaxAMPercentageLimit from UI to keep the improvement what we have done as part of YARN-4304.

          But after seeing the findbug warning, essentially we are boxing and unboxing this float variable. And I could see that there are some other parrellel effort going on in other tickets to avoid such cases.YARN-4630.

          So I think we can still keep float and avoid this problem. But we will loose small part of improvement there. So I think may be we can group this new metrics (maxAMPercentageLimit etc) in other DAO object and can use it. If its fine we can go with v1 patch here and unblock the scheduler issue. And other improvement can be tracked separately. Wangda Tan, Could you pls share your thoughts here.

          Show
          sunilg Sunil G added a comment - I think this has gone stale. Lets move forward. In YARN-4304 , one of the small improvement was to hide maxAMPercentageLimit for parent queue. We achieved this by declaring this variable as Float and was setting to null if queue is ParentQueue. Now we have got this corner scenario when labels are available in cluster, and no label-mappings are defined in CS. We have fixed partially in YARN-4634 , however we can still have some more cases like here. So ideally its better to handle null check for getMaxAMPercentageLimit from UI to keep the improvement what we have done as part of YARN-4304 . But after seeing the findbug warning, essentially we are boxing and unboxing this float variable. And I could see that there are some other parrellel effort going on in other tickets to avoid such cases. YARN-4630 . So I think we can still keep float and avoid this problem. But we will loose small part of improvement there. So I think may be we can group this new metrics (maxAMPercentageLimit etc) in other DAO object and can use it. If its fine we can go with v1 patch here and unblock the scheduler issue. And other improvement can be tracked separately. Wangda Tan , Could you pls share your thoughts here.
          Hide
          sunilg Sunil G added a comment -

          I think we are purposefully doing this for REST improvement. Looping Wangda Tan, we have tried to use boxed type Float for maxAMPercentageLimit to hide this unit for ParentQueue. Pls share your thoughts also.

          Show
          sunilg Sunil G added a comment - I think we are purposefully doing this for REST improvement. Looping Wangda Tan , we have tried to use boxed type Float for maxAMPercentageLimit to hide this unit for ParentQueue. Pls share your thoughts also.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Brahma Reddy Battula can you check findbugs warnings?

          Show
          rohithsharma Rohith Sharma K S added a comment - Brahma Reddy Battula can you check findbugs warnings?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 12s trunk passed
          +1 compile 0m 28s trunk passed with JDK v1.8.0_72
          +1 compile 0m 32s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 0m 36s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 5s trunk passed
          +1 javadoc 0m 22s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 28s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 31s the patch passed
          +1 compile 0m 24s the patch passed with JDK v1.8.0_72
          +1 javac 0m 24s the patch passed
          +1 compile 0m 27s the patch passed with JDK v1.7.0_95
          +1 javac 0m 27s the patch passed
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 0m 33s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 20s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 26s the patch passed with JDK v1.7.0_95
          -1 unit 70m 26s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72.
          -1 unit 71m 35s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          159m 18s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderQueueCapacityInfo(ResponseInfo, String) At CapacitySchedulerPage.java:then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderQueueCapacityInfo(ResponseInfo, String) At CapacitySchedulerPage.java:[line 155]
          JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790103/YARN-4624-003.patch
          JIRA Issue YARN-4624
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 512ae622e8cb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d7fdec1
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/10649/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10649/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10649/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10649/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10649/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10649/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10649/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 12s trunk passed +1 compile 0m 28s trunk passed with JDK v1.8.0_72 +1 compile 0m 32s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 22s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 28s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 31s the patch passed +1 compile 0m 24s the patch passed with JDK v1.8.0_72 +1 javac 0m 24s the patch passed +1 compile 0m 27s the patch passed with JDK v1.7.0_95 +1 javac 0m 27s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 20s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 26s the patch passed with JDK v1.7.0_95 -1 unit 70m 26s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72. -1 unit 71m 35s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 159m 18s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderQueueCapacityInfo(ResponseInfo, String) At CapacitySchedulerPage.java:then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderQueueCapacityInfo(ResponseInfo, String) At CapacitySchedulerPage.java: [line 155] JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790103/YARN-4624-003.patch JIRA Issue YARN-4624 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 512ae622e8cb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d7fdec1 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/10649/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/10649/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10649/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10649/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10649/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10649/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10649/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Cancelling and uploading the patch for jenkins..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Cancelling and uploading the patch for jenkins..
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Triggered jenkins manually

          Show
          rohithsharma Rohith Sharma K S added a comment - Triggered jenkins manually
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          +1 LGTM, pending jenkins

          Show
          rohithsharma Rohith Sharma K S added a comment - +1 LGTM, pending jenkins
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          As doing null check at capacities.getMaxAMLimitPercentage() == null changed to wrapper which was not used anywhere. and couldn't be checked null in PartitionQueueCapacitiesInfo#getMaxAMLimitPercentage (REST API call will fail).

          Show
          brahmareddy Brahma Reddy Battula added a comment - As doing null check at capacities.getMaxAMLimitPercentage() == null changed to wrapper which was not used anywhere. and couldn't be checked null in PartitionQueueCapacitiesInfo#getMaxAMLimitPercentage (REST API call will fail).
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          thanks Brahma Reddy Battula for providing patch.
          one nit: why are we using wrapper Float instead of primitive float?

          Show
          rohithsharma Rohith Sharma K S added a comment - thanks Brahma Reddy Battula for providing patch. one nit: why are we using wrapper Float instead of primitive float?
          Hide
          sunilg Sunil G added a comment -

          Uploaded a screen shot from my test cluster where I removed label-mapping from CS xml but kept few labels in cluster.

          Show
          sunilg Sunil G added a comment - Uploaded a screen shot from my test cluster where I removed label-mapping from CS xml but kept few labels in cluster.
          Hide
          sunilg Sunil G added a comment -

          Hi Brahma Reddy Battula
          Thanks for updating the patch. Latest patch looks good for me. +1
          I tested this patch and REST o/p is showing maxAMPercentage only for LeafQueue as expected. Also UI coming up correctly.

          Show
          sunilg Sunil G added a comment - Hi Brahma Reddy Battula Thanks for updating the patch. Latest patch looks good for me. +1 I tested this patch and REST o/p is showing maxAMPercentage only for LeafQueue as expected. Also UI coming up correctly.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Uploaded the patch ,kindly Review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch ,kindly Review..
          Hide
          sunilg Sunil G added a comment -

          Yes.

          Don't we need to check for null instead of 0 here?

          I think we can check for null, which looks more better here. For this, we can change the return type of capacities.getMaxAMLimitPercentage to Float.

          Show
          sunilg Sunil G added a comment - Yes. Don't we need to check for null instead of 0 here? I think we can check for null , which looks more better here. For this, we can change the return type of capacities.getMaxAMLimitPercentage to Float .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 27s trunk passed
          +1 compile 0m 26s trunk passed with JDK v1.8.0_72
          +1 compile 0m 29s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 0m 34s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 3s trunk passed
          +1 javadoc 0m 20s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 25s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 29s the patch passed
          +1 compile 0m 23s the patch passed with JDK v1.8.0_72
          +1 javac 0m 23s the patch passed
          +1 compile 0m 27s the patch passed with JDK v1.7.0_95
          +1 javac 0m 27s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 32s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 13s the patch passed
          +1 javadoc 0m 18s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95
          -1 unit 70m 54s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72.
          -1 unit 72m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          159m 15s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787067/YARN-2674-002.patch
          JIRA Issue YARN-4624
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 18b0d45ceb1b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fa00d3e
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10556/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10556/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10556/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10556/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10556/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10556/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 27s trunk passed +1 compile 0m 26s trunk passed with JDK v1.8.0_72 +1 compile 0m 29s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 3s trunk passed +1 javadoc 0m 20s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 25s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 29s the patch passed +1 compile 0m 23s the patch passed with JDK v1.8.0_72 +1 javac 0m 23s the patch passed +1 compile 0m 27s the patch passed with JDK v1.7.0_95 +1 javac 0m 27s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 13s the patch passed +1 javadoc 0m 18s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95 -1 unit 70m 54s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72. -1 unit 72m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 159m 15s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787067/YARN-2674-002.patch JIRA Issue YARN-4624 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 18b0d45ceb1b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fa00d3e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10556/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10556/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10556/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10556/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10556/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-YARN-Build/10556/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          devaraj.k Devaraj K added a comment -

          Thanks Brahma Reddy Battula for the updated patch.

          +          capacities.getMaxAMLimitPercentage() == 0
          +             ? 0 : capacities.getMaxAMLimitPercentage())).
          

          Don't we need to check for null instead of 0 here? Please verify the scenario with the patch changes.

          Show
          devaraj.k Devaraj K added a comment - Thanks Brahma Reddy Battula for the updated patch. + capacities.getMaxAMLimitPercentage() == 0 + ? 0 : capacities.getMaxAMLimitPercentage())). Don't we need to check for null instead of 0 here? Please verify the scenario with the patch changes.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Sunil G and Devaraj K kindly review the patch.. thanks..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Sunil G and Devaraj K kindly review the patch.. thanks..
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Sunil G thanks for your comment..Yes, null check is enough.. Uploaded the patch..kindly review..Devaraj K as change is in UI, not added the testcase..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Sunil G thanks for your comment..Yes, null check is enough.. Uploaded the patch..kindly review.. Devaraj K as change is in UI, not added the testcase..
          Hide
          sunilg Sunil G added a comment -

          Hi Devaraj K
          Current patch will revert one optimization we have done in YARN-4304. we wanted to avoid showing maxAmLimitPercentage to all parent queues in REST o/p. So we wanted to pass null for parent queues and current am percentage for leaf queue. Hence we used a boxed type variable. So cud we handle null check in CapacitySchedulerPage. Pls share your thoughts.

          Show
          sunilg Sunil G added a comment - Hi Devaraj K Current patch will revert one optimization we have done in YARN-4304 . we wanted to avoid showing maxAmLimitPercentage to all parent queues in REST o/p. So we wanted to pass null for parent queues and current am percentage for leaf queue. Hence we used a boxed type variable. So cud we handle null check in CapacitySchedulerPage. Pls share your thoughts.
          Hide
          devaraj.k Devaraj K added a comment -

          Thanks Brahma Reddy Battula for reporting and providing patch.

          Would you mind adding a test for this as part of the patch?

          Show
          devaraj.k Devaraj K added a comment - Thanks Brahma Reddy Battula for reporting and providing patch. Would you mind adding a test for this as part of the patch?
          Hide
          sunilg Sunil G added a comment -

          Offline synced with Naga and got the scenario:

          • Node labels are enabled in cluster
          • Also few labels are added to cluster from CLI.
          • But Queue-Label Mapping is NOT configured in capacity-scheduler.xml

          Since queue mapping is not configured, call flow is not happening like with/with out labels. Hence cause this problem. Nice catch, and good scenario!
          So As mentioned in above comments, we can have a null check in CapacitySchedulerPage and this can be resolved. I verified and its looking fine.

          Show
          sunilg Sunil G added a comment - Offline synced with Naga and got the scenario: Node labels are enabled in cluster Also few labels are added to cluster from CLI. But Queue-Label Mapping is NOT configured in capacity-scheduler.xml Since queue mapping is not configured, call flow is not happening like with/with out labels. Hence cause this problem. Nice catch, and good scenario! So As mentioned in above comments, we can have a null check in CapacitySchedulerPage and this can be resolved. I verified and its looking fine.
          Hide
          sunilg Sunil G added a comment -

          maxAMLimitPercentage is set to null only in case of ParentQueue. For LeafQueue, maxAMLimitPercentage will correctly set and UI displays only LeafQueue's capacity. So pls help to share more information abt this problem. (logs/config details).

          Show
          sunilg Sunil G added a comment - maxAMLimitPercentage is set to null only in case of ParentQueue. For LeafQueue, maxAMLimitPercentage will correctly set and UI displays only LeafQueue's capacity. So pls help to share more information abt this problem. (logs/config details).
          Hide
          sunilg Sunil G added a comment -

          Hi Brahma Reddy Battula
          I have installed trunk and tested UI with labels and w/o labels. I could not see this problem while accessing Scheduler UI. Could you please help to share the config/version where this issue happened.

          Show
          sunilg Sunil G added a comment - Hi Brahma Reddy Battula I have installed trunk and tested UI with labels and w/o labels. I could not see this problem while accessing Scheduler UI. Could you please help to share the config/version where this issue happened.
          Hide
          sunilg Sunil G added a comment -

          Please discard my prior suggestion, it will still carry maxAMLimitPercentage in REST o/p when not used.

          So we can have this check in CapacitySchedulerPage itself. Could we have a null check here and then display 0 in that case. And both cases can be handled then.

          _("Configured Max Application Master Limit:", StringUtils.format("%.1f",
                    capacities.getMaxAMLimitPercentage())).
          
          Show
          sunilg Sunil G added a comment - Please discard my prior suggestion, it will still carry maxAMLimitPercentage in REST o/p when not used. So we can have this check in CapacitySchedulerPage itself. Could we have a null check here and then display 0 in that case. And both cases can be handled then. _( "Configured Max Application Master Limit:" , StringUtils.format( "%.1f" , capacities.getMaxAMLimitPercentage())).
          Hide
          sunilg Sunil G added a comment -

          Hi Brahma Reddy Battula
          I suggest we can do like below in PartitionQueueCapacitiesInfo to keep REST o/p improvement.

            public float getMaxAMLimitPercentage() {
              return (maxAMLimitPercentage == null) ? 0f : maxAMLimitPercentage;
            }
          

          cc/ Wangda Tan

          Show
          sunilg Sunil G added a comment - Hi Brahma Reddy Battula I suggest we can do like below in PartitionQueueCapacitiesInfo to keep REST o/p improvement. public float getMaxAMLimitPercentage() { return (maxAMLimitPercentage == null ) ? 0f : maxAMLimitPercentage; } cc/ Wangda Tan
          Hide
          sunilg Sunil G added a comment -

          Oh. Thanks for raising and for the patch. We used Float for REST o/p handling. I think we must handle same in UI also. Let me try seeing CapacitySchedulerPage.

          Show
          sunilg Sunil G added a comment - Oh. Thanks for raising and for the patch. We used Float for REST o/p handling. I think we must handle same in UI also. Let me try seeing CapacitySchedulerPage.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Because of auto-boxing for maxAMLimitPercentage..Attached the patch kindly review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Because of auto-boxing for maxAMLimitPercentage ..Attached the patch kindly review..

            People

            • Assignee:
              brahmareddy Brahma Reddy Battula
              Reporter:
              brahmareddy Brahma Reddy Battula
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development