Hadoop YARN
  1. Hadoop YARN
  2. YARN-429

capacity-scheduler config missing from yarn-test artifact

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 2.0.4-alpha
    • Component/s: resourcemanager
    • Labels:
      None

      Description

      MiniYARNCluster and MiniMRCluster are unusable by downstream projects with the 2.0.3-alpha release, since the capacity-scheduler configuration is missing from the test artifact.
      hadoop-yarn-server-tests-3.0.0-SNAPSHOT-tests.jar should include the default capacity-scheduler configuration. Also, this doesn't need to be part of the default classpath - and should be moved out of the top level directory in the dist package.

      1. YARN-429.txt
        5 kB
        Siddharth Seth

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1365 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1365/)
        YARN-429. capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501)

        Result = SUCCESS
        hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501
        Files :

        • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1365 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1365/ ) YARN-429 . capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1337 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1337/)
        YARN-429. capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501)

        Result = SUCCESS
        hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501
        Files :

        • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1337 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1337/ ) YARN-429 . capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #148 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/148/)
        YARN-429. capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501)

        Result = SUCCESS
        hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501
        Files :

        • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #148 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/148/ ) YARN-429 . capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
        Hide
        stack added a comment -

        Hitesh Shah nvm. just saw the trunk commit... ignore above. Thanks.

        Show
        stack added a comment - Hitesh Shah nvm. just saw the trunk commit... ignore above. Thanks.
        Hide
        stack added a comment -

        Hitesh Shah May we have it in 2.0 hadoop too please?

        Show
        stack added a comment - Hitesh Shah May we have it in 2.0 hadoop too please?
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3425 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3425/)
        YARN-429. capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501)

        Result = SUCCESS
        hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501
        Files :

        • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3425 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3425/ ) YARN-429 . capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
        Hide
        Hitesh Shah added a comment -

        Committed to branch-2 and trunk. Thanks Sid.

        Show
        Hitesh Shah added a comment - Committed to branch-2 and trunk. Thanks Sid.
        Hide
        Hitesh Shah added a comment -

        +1. Looks good. committing shortly.

        Show
        Hitesh Shah added a comment - +1. Looks good. committing shortly.
        Hide
        Andrew Purtell added a comment -

        Thanks for looking into this. +1 on patch.

        Show
        Andrew Purtell added a comment - Thanks for looking into this. +1 on patch.
        Hide
        stack added a comment -

        Patch looks reasonable to me.

        Show
        stack added a comment - Patch looks reasonable to me.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12571045/YARN-429.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 tests included appear to have a timeout.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/454//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/454//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571045/YARN-429.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/454//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/454//console This message is automatically generated.
        Hide
        Siddharth Seth added a comment -

        Tested the patch locally with help from Ted Yu.

        Ran TestTableMapReduce and TestHFileOutputFormat against hadoop-trunk with and without the updated patch. The tests did fail in both runs, but the error was not related to the capacity-scheduler after applying the patch.

        Show
        Siddharth Seth added a comment - Tested the patch locally with help from Ted Yu . Ran TestTableMapReduce and TestHFileOutputFormat against hadoop-trunk with and without the updated patch. The tests did fail in both runs, but the error was not related to the capacity-scheduler after applying the patch.
        Hide
        Arun C Murthy added a comment -

        This might have been caused by YARN-140, investigating further.

        Show
        Arun C Murthy added a comment - This might have been caused by YARN-140 , investigating further.
        Hide
        Siddharth Seth added a comment -

        Patch to include capacity-scheduler.xml in hadoop-yarn-server-tests-$

        {version}

        -tests.jar

        Also changes the dist packaging to move this file under 'share/hadoop/yarn/test' instead of 'share/hadoop/yarn'.

        Is there a standard location for test-jars - instead of share/hadoop/yarn/test ?

        Show
        Siddharth Seth added a comment - Patch to include capacity-scheduler.xml in hadoop-yarn-server-tests-$ {version} -tests.jar Also changes the dist packaging to move this file under 'share/hadoop/yarn/test' instead of 'share/hadoop/yarn'. Is there a standard location for test-jars - instead of share/hadoop/yarn/test ?
        Hide
        Siddharth Seth added a comment -

        stack trace

        org.apache.hadoop.yarn.YarnException: Failed to Start org.apache.hadoop.mapred.MiniMRCluster
                at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78)
                at org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:75)
                at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:180)
                at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:170)
                at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:162)
                at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:123)
                at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniMapReduceCluster(HBaseTestingUtility.java:1574)
        ...
        Caused by: org.apache.hadoop.yarn.YarnException: java.lang.IllegalStateException: Queue configuration missing child queue names for root
                at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:181)
                at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
                ... 38 more
        Caused by: java.lang.IllegalStateException: Queue configuration missing child queue names for root
                at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:328)
                at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:255)
                at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:220)
                at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.init(ResourceManager.java:226)
                at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:161)
        
        Show
        Siddharth Seth added a comment - stack trace org.apache.hadoop.yarn.YarnException: Failed to Start org.apache.hadoop.mapred.MiniMRCluster at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78) at org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:75) at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:180) at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:170) at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:162) at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:123) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniMapReduceCluster(HBaseTestingUtility.java:1574) ... Caused by: org.apache.hadoop.yarn.YarnException: java.lang.IllegalStateException: Queue configuration missing child queue names for root at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:181) at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) ... 38 more Caused by: java.lang.IllegalStateException: Queue configuration missing child queue names for root at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:328) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:255) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:220) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.init(ResourceManager.java:226) at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:161)

          People

          • Assignee:
            Siddharth Seth
            Reporter:
            Siddharth Seth
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development