Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3894

RM startup should fail for wrong CS xml NodeLabel capacity configuration

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Currently in capacity Scheduler when capacity configuration is wrong
      RM will shutdown, but not incase of NodeLabels capacity mismatch

      In CapacityScheduler#initializeQueues

        private void initializeQueues(CapacitySchedulerConfiguration conf)
          throws IOException {   
          root = 
              parseQueue(this, conf, null, CapacitySchedulerConfiguration.ROOT, 
                  queues, queues, noop);
          labelManager.reinitializeQueueLabels(getQueueToLabels());
          root = 
              parseQueue(this, conf, null, CapacitySchedulerConfiguration.ROOT, 
                  queues, queues, noop);
          LOG.info("Initialized root queue " + root);
          initializeQueueMappings();
          setQueueAcls(authorizer, queues);
        }
      

      labelManager is initialized from queues and calculation for Label level capacity mismatch happens in parseQueue . So during initialization parseQueue the labels will be empty .

      Steps to reproduce

      1. Configure RM with capacity scheduler
      2. Add one or two node label from rmadmin
      3. Configure capacity xml with nodelabel but issue with capacity configuration for already added label
      4. Restart both RM
      5. Check on service init of capacity scheduler node label list is populated

      Expected

      RM should not start

      Current exception on reintialize check

      2015-07-07 19:18:25,655 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: default: capacity=0.5, absoluteCapacity=0.5, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0
      2015-07-07 19:18:25,656 WARN org.apache.hadoop.yarn.server.resourcemanager.AdminService: Exception refresh queues.
      java.io.IOException: Failed to re-init queues
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:383)
              at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:376)
              at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:605)
              at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:314)
              at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
              at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:824)
              at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:420)
              at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
              at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
      Caused by: java.lang.IllegalArgumentException: Illegal capacity of 0.5 for children of queue root for label=node2
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setChildQueues(ParentQueue.java:159)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:639)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:503)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:379)
              ... 8 more
      2015-07-07 19:18:25,656 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf   OPERATION=refreshQueues TARGET=AdminService     RESULT=FAILURE  DESCRIPTION=Exception refresh queues.   PERMISSIONS=
      2015-07-07 19:18:25,656 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf   OPERATION=transitionToActive    TARGET=RMHAProtocolService      RESULT=FAILURE  DESCRIPTION=Exception transitioning to active   PERMISSIONS=
      2015-07-07 19:18:25,656 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election
      org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
              at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:128)
              at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:824)
              at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:420)
              at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
              at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
      Caused by: org.apache.hadoop.ha.ServiceFailedException: Error when transitioning to Active mode
              at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:321)
              at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
              ... 4 more
      Caused by: org.apache.hadoop.ha.ServiceFailedException: java.io.IOException: Failed to re-init queues
              at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:617)
              at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:314)
              ... 5 more
      
      
      1. 0001-YARN-3894.patch
        4 kB
        Bibin A Chundatt
      2. 0002-YARN-3894.patch
        4 kB
        Bibin A Chundatt
      3. capacity-scheduler.xml
        2 kB
        Bibin A Chundatt

        Issue Links

          Activity

          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Wangda Tan . Should we reinitialize queue again for capacity validation?

          Show
          bibinchundatt Bibin A Chundatt added a comment - Wangda Tan . Should we reinitialize queue again for capacity validation?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching conf used for capacity-scheduler.xml . On Serviceinit the queue is not failing but on reinitialize the same is failing. yarn-site added in YARN-3893

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching conf used for capacity-scheduler.xml . On Serviceinit the queue is not failing but on reinitialize the same is failing. yarn-site added in YARN-3893
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Analysis Detail:

          1. During CapacityScheduler#serviceInit the value of labelManager.getClusterNodeLabelNames() will be empty. While setting ParentQueue#setChildQueues validation is not done for Node Label Capacity
          Show
          bibinchundatt Bibin A Chundatt added a comment - Analysis Detail: During CapacityScheduler#serviceInit the value of labelManager.getClusterNodeLabelNames() will be empty. While setting ParentQueue#setChildQueues validation is not done for Node Label Capacity
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Detailed analysis and root cause

          1. Capacity scheduler queue initialization happens Capacity#serviceInit
          2. RMNodeLabelsManager#addToCluserNodeLabels store is added in service start on recovery
          Show
          bibinchundatt Bibin A Chundatt added a comment - Detailed analysis and root cause Capacity scheduler queue initialization happens Capacity#serviceInit RMNodeLabelsManager#addToCluserNodeLabels store is added in service start on recovery
          Hide
          sunilg Sunil G added a comment -

          Thanks Bibin A Chundatt for reporting and providing analysis.

          During initScheduler call from CapacityScheduler#serviceInit, we will initialize the queues. In the same callflow, we also will validate the capacity of nodelabel against the queue capacity from ParentQueue#setChildQueues.

             // check label capacities
              for (String nodeLabel : labelManager.getClusterNodeLabelNames()) {
                float capacityByLabel = queueCapacities.getCapacity(nodeLabel);
                // check children's labels
                float sum = 0;
                for (CSQueue queue : childQueues) {
                  sum += queue.getQueueCapacities().getCapacity(nodeLabel);
                }
                if ((capacityByLabel > 0 && Math.abs(1.0f - sum) > PRECISION)
                    || (capacityByLabel == 0) && (sum > 0)) {
                  throw new IllegalArgumentException("Illegal" + " capacity of "
                      + sum + " for children of queue " + queueName
                      + " for label=" + nodeLabel);
                }
              }
          

          As per this code, if there is a mismatch in capacity for nodelabel against the queue capacity, it should through IllegalArgumentException. But this will not happen in a case where we configure a wrong capacity for label in cs xml, and restart RM.

          Issue:
          During CommonNodeLabelsManager#serviceStart, labels will re-populated from old mirror file. But initScheduler and above call flow will happen from serviceInit instead of serviceStart
          This will make labelManager.getClusterNodeLabelNames() call as empty in above code. and desired exception wont be thrown.

          IMO We can move the node label init and recovery to serviceInit rather than serviceStart. Wangda Tan, could you please pool in your thoughts.

          Show
          sunilg Sunil G added a comment - Thanks Bibin A Chundatt for reporting and providing analysis. During initScheduler call from CapacityScheduler#serviceInit , we will initialize the queues. In the same callflow, we also will validate the capacity of nodelabel against the queue capacity from ParentQueue#setChildQueues . // check label capacities for ( String nodeLabel : labelManager.getClusterNodeLabelNames()) { float capacityByLabel = queueCapacities.getCapacity(nodeLabel); // check children's labels float sum = 0; for (CSQueue queue : childQueues) { sum += queue.getQueueCapacities().getCapacity(nodeLabel); } if ((capacityByLabel > 0 && Math .abs(1.0f - sum) > PRECISION) || (capacityByLabel == 0) && (sum > 0)) { throw new IllegalArgumentException( "Illegal" + " capacity of " + sum + " for children of queue " + queueName + " for label=" + nodeLabel); } } As per this code, if there is a mismatch in capacity for nodelabel against the queue capacity, it should through IllegalArgumentException . But this will not happen in a case where we configure a wrong capacity for label in cs xml, and restart RM. Issue: During CommonNodeLabelsManager#serviceStart , labels will re-populated from old mirror file. But initScheduler and above call flow will happen from serviceInit instead of serviceStart This will make labelManager.getClusterNodeLabelNames() call as empty in above code. and desired exception wont be thrown. IMO We can move the node label init and recovery to serviceInit rather than serviceStart. Wangda Tan , could you please pool in your thoughts.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Hi Bibin A Chundatt,
          Are you using latest trunk? In latest trunk, node label related capacity checking for capacity scheduler is not related to node label manager initialization. Misconfiguration of node label capacity should fail CS.

          Show
          leftnoteasy Wangda Tan added a comment - Hi Bibin A Chundatt , Are you using latest trunk? In latest trunk, node label related capacity checking for capacity scheduler is not related to node label manager initialization. Misconfiguration of node label capacity should fail CS.
          Hide
          leftnoteasy Wangda Tan added a comment -

          After read comment from Sunil G, I know what's the problem. In ParentQueue.setChilQueues, using QueueCapacities.getExistingNodeLabels instead of labelManager.getClusterNodeLabels should be able to solve this problem.

          Thoughts? Sunil G, Bibin A Chundatt.

          Show
          leftnoteasy Wangda Tan added a comment - After read comment from Sunil G , I know what's the problem. In ParentQueue.setChilQueues, using QueueCapacities.getExistingNodeLabels instead of labelManager.getClusterNodeLabels should be able to solve this problem. Thoughts? Sunil G , Bibin A Chundatt .
          Hide
          sunilg Sunil G added a comment -

          HI Wangda Tan
          Thank you for sharing your thoughts.
          +1 for using QueueCapacities.getExistingNodeLabels. It will solve the problem.

          Show
          sunilg Sunil G added a comment - HI Wangda Tan Thank you for sharing your thoughts. +1 for using QueueCapacities.getExistingNodeLabels. It will solve the problem.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Wangda Tan
          Thank you for sharing your thoughts.
          Will upload the patch soon.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Wangda Tan Thank you for sharing your thoughts. Will upload the patch soon.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attached patch as per discussion.
          Please review patch

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attached patch as per discussion. Please review patch
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Sunil G and Wangda Tan Please review patch

          Show
          bibinchundatt Bibin A Chundatt added a comment - Sunil G and Wangda Tan Please review patch
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 5s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 42s There were no new javac warning messages.
          +1 javadoc 9m 41s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 46s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 yarn tests 51m 25s Tests failed in hadoop-yarn-server-resourcemanager.
              89m 23s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.TestApplicationCleanup
            hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12744898/0001-YARN-3894.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 1df39c1
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8509/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8509/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8509/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 42s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 46s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 51m 25s Tests failed in hadoop-yarn-server-resourcemanager.     89m 23s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.TestApplicationCleanup   hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744898/0001-YARN-3894.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1df39c1 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8509/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8509/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8509/console This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Testcase failures are not due of this patch to my knowledge. Please check

          Show
          bibinchundatt Bibin A Chundatt added a comment - Testcase failures are not due of this patch to my knowledge. Please check
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Bibin A Chundatt working on the patch.

          YARN-3909 YARN-3910 YARN-3918 are tracking failure tests, not related to this patch.

          One minor comment on the test: could you move the test to TestQueueParsing? And please verify test fails without the change and passed with the change.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Bibin A Chundatt working on the patch. YARN-3909 YARN-3910 YARN-3918 are tracking failure tests, not related to this patch. One minor comment on the test: could you move the test to TestQueueParsing? And please verify test fails without the change and passed with the change.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Wangda Tan patch for review after handling comments
          After fix

          2015-07-12 23:37:51,101 INFO  [Thread-2] service.AbstractService (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in state INITED; cause: java.lang.IllegalArgumentException: Illegal capacity of 0.7 for children of queue root for label=z
          java.lang.IllegalArgumentException: Illegal capacity of 0.7 for children of queue root for label=z
          	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setChildQueues(ParentQueue.java:159)
          	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:639)
          	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:490)
          	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:319)
          	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:349)
          	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
          	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
          	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:559)
          	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
          	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:954)
          	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:254)
          	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
          	at org.apache.hadoop.yarn.server.resourcemanager.MockRM.<init>(MockRM.java:119)
          	at org.apache.hadoop.yarn.server.resourcemanager.MockRM.<init>(MockRM.java:112)
          	at org.apache.hadoop.yarn.server.resourcemanager.MockRM.<init>(MockRM.java:108)
          	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing$1.<init>(TestQueueParsing.java:929)
          	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing.testRMStartWrongNodeCapacity(TestQueueParsing.java:929)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:606)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          
          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Wangda Tan patch for review after handling comments After fix 2015-07-12 23:37:51,101 INFO [ Thread -2] service.AbstractService (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in state INITED; cause: java.lang.IllegalArgumentException: Illegal capacity of 0.7 for children of queue root for label=z java.lang.IllegalArgumentException: Illegal capacity of 0.7 for children of queue root for label=z at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setChildQueues(ParentQueue.java:159) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:639) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:490) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:319) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:349) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:559) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:954) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:254) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.MockRM.<init>(MockRM.java:119) at org.apache.hadoop.yarn.server.resourcemanager.MockRM.<init>(MockRM.java:112) at org.apache.hadoop.yarn.server.resourcemanager.MockRM.<init>(MockRM.java:108) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing$1.<init>(TestQueueParsing.java:929) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing.testRMStartWrongNodeCapacity(TestQueueParsing.java:929) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 59s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 39s There were no new javac warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 46s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 21s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 yarn tests 51m 19s Tests failed in hadoop-yarn-server-resourcemanager.
              89m 2s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates
            hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions
            hadoop.yarn.server.resourcemanager.TestApplicationCleanup
            hadoop.yarn.server.resourcemanager.TestResourceTrackerService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12744946/0002-YARN-3894.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d7319de
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8514/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8514/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8514/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 59s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 39s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 46s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 21s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 51m 19s Tests failed in hadoop-yarn-server-resourcemanager.     89m 2s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates   hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions   hadoop.yarn.server.resourcemanager.TestApplicationCleanup   hadoop.yarn.server.resourcemanager.TestResourceTrackerService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744946/0002-YARN-3894.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d7319de hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8514/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8514/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8514/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Committed to trunk/branch-2, thanks Bibin A Chundatt and review from Sunil G!

          Show
          leftnoteasy Wangda Tan added a comment - Committed to trunk/branch-2, thanks Bibin A Chundatt and review from Sunil G !
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8154 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8154/)
          YARN-3894. RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda) (wangda: rev 5ed1fead6b5ec24bb0ce1a3db033c2ee1ede4bb4)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8154 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8154/ ) YARN-3894 . RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda) (wangda: rev 5ed1fead6b5ec24bb0ce1a3db033c2ee1ede4bb4) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Sunil G and Wangda Tan thanks a lot for review and commit

          Show
          bibinchundatt Bibin A Chundatt added a comment - Sunil G and Wangda Tan thanks a lot for review and commit
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #255 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/255/)
          YARN-3894. RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda) (wangda: rev 5ed1fead6b5ec24bb0ce1a3db033c2ee1ede4bb4)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #255 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/255/ ) YARN-3894 . RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda) (wangda: rev 5ed1fead6b5ec24bb0ce1a3db033c2ee1ede4bb4) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #985 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/985/)
          YARN-3894. RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda) (wangda: rev 5ed1fead6b5ec24bb0ce1a3db033c2ee1ede4bb4)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #985 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/985/ ) YARN-3894 . RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda) (wangda: rev 5ed1fead6b5ec24bb0ce1a3db033c2ee1ede4bb4) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #243 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/243/)
          YARN-3894. RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda) (wangda: rev 5ed1fead6b5ec24bb0ce1a3db033c2ee1ede4bb4)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #243 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/243/ ) YARN-3894 . RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda) (wangda: rev 5ed1fead6b5ec24bb0ce1a3db033c2ee1ede4bb4) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2182 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2182/)
          YARN-3894. RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda) (wangda: rev 5ed1fead6b5ec24bb0ce1a3db033c2ee1ede4bb4)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2182 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2182/ ) YARN-3894 . RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda) (wangda: rev 5ed1fead6b5ec24bb0ce1a3db033c2ee1ede4bb4) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2201 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2201/)
          YARN-3894. RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda) (wangda: rev 5ed1fead6b5ec24bb0ce1a3db033c2ee1ede4bb4)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2201 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2201/ ) YARN-3894 . RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda) (wangda: rev 5ed1fead6b5ec24bb0ce1a3db033c2ee1ede4bb4) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #253 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/253/)
          YARN-3894. RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda) (wangda: rev 5ed1fead6b5ec24bb0ce1a3db033c2ee1ede4bb4)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #253 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/253/ ) YARN-3894 . RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda) (wangda: rev 5ed1fead6b5ec24bb0ce1a3db033c2ee1ede4bb4) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java

            People

            • Assignee:
              bibinchundatt Bibin A Chundatt
              Reporter:
              bibinchundatt Bibin A Chundatt
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development