Hadoop YARN
  1. Hadoop YARN
  2. YARN-429

capacity-scheduler config missing from yarn-test artifact

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 2.0.4-alpha
    • Component/s: resourcemanager
    • Labels:
      None

      Description

      MiniYARNCluster and MiniMRCluster are unusable by downstream projects with the 2.0.3-alpha release, since the capacity-scheduler configuration is missing from the test artifact.
      hadoop-yarn-server-tests-3.0.0-SNAPSHOT-tests.jar should include the default capacity-scheduler configuration. Also, this doesn't need to be part of the default classpath - and should be moved out of the top level directory in the dist package.

      1. YARN-429.txt
        5 kB
        Siddharth Seth

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1365 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1365/)
          YARN-429. capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501)

          Result = SUCCESS
          hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1365 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1365/ ) YARN-429 . capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1337 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1337/)
          YARN-429. capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501)

          Result = SUCCESS
          hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1337 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1337/ ) YARN-429 . capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #148 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/148/)
          YARN-429. capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501)

          Result = SUCCESS
          hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #148 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/148/ ) YARN-429 . capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
          Hide
          stack added a comment -

          Hitesh Shah nvm. just saw the trunk commit... ignore above. Thanks.

          Show
          stack added a comment - Hitesh Shah nvm. just saw the trunk commit... ignore above. Thanks.
          Hide
          stack added a comment -

          Hitesh Shah May we have it in 2.0 hadoop too please?

          Show
          stack added a comment - Hitesh Shah May we have it in 2.0 hadoop too please?
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3425 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3425/)
          YARN-429. capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501)

          Result = SUCCESS
          hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3425 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3425/ ) YARN-429 . capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1453501 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml
          Hide
          Hitesh Shah added a comment -

          Committed to branch-2 and trunk. Thanks Sid.

          Show
          Hitesh Shah added a comment - Committed to branch-2 and trunk. Thanks Sid.
          Hide
          Hitesh Shah added a comment -

          +1. Looks good. committing shortly.

          Show
          Hitesh Shah added a comment - +1. Looks good. committing shortly.
          Hide
          Andrew Purtell added a comment -

          Thanks for looking into this. +1 on patch.

          Show
          Andrew Purtell added a comment - Thanks for looking into this. +1 on patch.
          Hide
          stack added a comment -

          Patch looks reasonable to me.

          Show
          stack added a comment - Patch looks reasonable to me.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12571045/YARN-429.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/454//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/454//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571045/YARN-429.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/454//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/454//console This message is automatically generated.
          Hide
          Siddharth Seth added a comment -

          Tested the patch locally with help from Ted Yu.

          Ran TestTableMapReduce and TestHFileOutputFormat against hadoop-trunk with and without the updated patch. The tests did fail in both runs, but the error was not related to the capacity-scheduler after applying the patch.

          Show
          Siddharth Seth added a comment - Tested the patch locally with help from Ted Yu . Ran TestTableMapReduce and TestHFileOutputFormat against hadoop-trunk with and without the updated patch. The tests did fail in both runs, but the error was not related to the capacity-scheduler after applying the patch.
          Hide
          Arun C Murthy added a comment -

          This might have been caused by YARN-140, investigating further.

          Show
          Arun C Murthy added a comment - This might have been caused by YARN-140 , investigating further.
          Hide
          Siddharth Seth added a comment -

          Patch to include capacity-scheduler.xml in hadoop-yarn-server-tests-$

          {version}

          -tests.jar

          Also changes the dist packaging to move this file under 'share/hadoop/yarn/test' instead of 'share/hadoop/yarn'.

          Is there a standard location for test-jars - instead of share/hadoop/yarn/test ?

          Show
          Siddharth Seth added a comment - Patch to include capacity-scheduler.xml in hadoop-yarn-server-tests-$ {version} -tests.jar Also changes the dist packaging to move this file under 'share/hadoop/yarn/test' instead of 'share/hadoop/yarn'. Is there a standard location for test-jars - instead of share/hadoop/yarn/test ?
          Hide
          Siddharth Seth added a comment -

          stack trace

          org.apache.hadoop.yarn.YarnException: Failed to Start org.apache.hadoop.mapred.MiniMRCluster
                  at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78)
                  at org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:75)
                  at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:180)
                  at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:170)
                  at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:162)
                  at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:123)
                  at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniMapReduceCluster(HBaseTestingUtility.java:1574)
          ...
          Caused by: org.apache.hadoop.yarn.YarnException: java.lang.IllegalStateException: Queue configuration missing child queue names for root
                  at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:181)
                  at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
                  ... 38 more
          Caused by: java.lang.IllegalStateException: Queue configuration missing child queue names for root
                  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:328)
                  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:255)
                  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:220)
                  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.init(ResourceManager.java:226)
                  at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:161)
          
          Show
          Siddharth Seth added a comment - stack trace org.apache.hadoop.yarn.YarnException: Failed to Start org.apache.hadoop.mapred.MiniMRCluster at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78) at org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:75) at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:180) at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:170) at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:162) at org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:123) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniMapReduceCluster(HBaseTestingUtility.java:1574) ... Caused by: org.apache.hadoop.yarn.YarnException: java.lang.IllegalStateException: Queue configuration missing child queue names for root at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:181) at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) ... 38 more Caused by: java.lang.IllegalStateException: Queue configuration missing child queue names for root at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:328) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:255) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:220) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.init(ResourceManager.java:226) at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:161)

            People

            • Assignee:
              Siddharth Seth
              Reporter:
              Siddharth Seth
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development