Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-514

Check for invalid queues in capacity scheduler

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The ResourceManagerConf class that is being moved to the capacity scheduler of HADOOP-3445 needs to check for a queue name that is not configured in the resource-manager-conf.xml file in its APIs and fail if it is not available. This feature was originally available, but due to subsequent changes in HADOOP-3698, queues are no longer being configured as part of the mentioned configuration file. Hence we need a different mechanism to check for valid queues.

      1. HADOOP-4079-1.patch
        5 kB
        Sreekanth Ramakrishnan
      2. HADOOP-4079-2.patch
        5 kB
        Sreekanth Ramakrishnan
      3. HADOOP-4079-3.patch
        5 kB
        Sreekanth Ramakrishnan
      4. HADOOP-4079-4.patch
        5 kB
        Sreekanth Ramakrishnan

        Issue Links

          Activity

          Hide
          Hemanth Yamijala added a comment -

          A simple proposal is to do what is done in HADOOP-3698. That is, we should parse the configuration at initialization and populate a mapping between queue names and the configured properties. This way, it will be easy to check if the map contains the queue name being requested in the APIs.

          Show
          Hemanth Yamijala added a comment - A simple proposal is to do what is done in HADOOP-3698 . That is, we should parse the configuration at initialization and populate a mapping between queue names and the configured properties. This way, it will be easy to check if the map contains the queue name being requested in the APIs.
          Hide
          Hemanth Yamijala added a comment -

          Considering how the code has turned out, the Capacity Scheduler iterates over queues it gets from the framework's QueueManager class. Now, when it asks the CapacitySchedulerConf for a property for this queue, if nothing is configured for this queue in capacity-scheduler.xml, a hard-coded default value is returned. This may not be what the administrator intended, because the defaults are not defined by him/her. Hence, this issue exists.

          HADOOP-4178 is trying to implement a mechanism to make these default values itself configurable. If it does that, then it would be a feature that if a property is not configured for a queue in the capacity scheduler's configuration, the configured default is used. This makes it very simple for the administrator to define new queues. And it would invalidate the need to check for invalid queues, as required by this issue.

          So in summary, if HADOOP-4178 is fixed, we can mark this bug as invalid. Comments ?

          Show
          Hemanth Yamijala added a comment - Considering how the code has turned out, the Capacity Scheduler iterates over queues it gets from the framework's QueueManager class. Now, when it asks the CapacitySchedulerConf for a property for this queue, if nothing is configured for this queue in capacity-scheduler.xml , a hard-coded default value is returned. This may not be what the administrator intended, because the defaults are not defined by him/her. Hence, this issue exists. HADOOP-4178 is trying to implement a mechanism to make these default values itself configurable. If it does that, then it would be a feature that if a property is not configured for a queue in the capacity scheduler's configuration, the configured default is used. This makes it very simple for the administrator to define new queues. And it would invalidate the need to check for invalid queues, as required by this issue. So in summary, if HADOOP-4178 is fixed, we can mark this bug as invalid. Comments ?
          Hide
          Owen O'Malley added a comment -

          I'd suggest that we warn in the JobTracker log at start up if the capacity scheduler configuration defines a queue that isn't present in the QueueManager. It will most likely be a typo on one side or the other. Not defining properties should be ok, since they will just fall back to the defaults defined via HADOOP-4178.

          Show
          Owen O'Malley added a comment - I'd suggest that we warn in the JobTracker log at start up if the capacity scheduler configuration defines a queue that isn't present in the QueueManager. It will most likely be a typo on one side or the other. Not defining properties should be ok, since they will just fall back to the defaults defined via HADOOP-4178 .
          Hide
          Sreekanth Ramakrishnan added a comment -

          Marking HADOOP-4178 as a related issue to this issue.

          Show
          Sreekanth Ramakrishnan added a comment - Marking HADOOP-4178 as a related issue to this issue.
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching patch with following changes:

          • Added
            CapacitySchedulerConf
             Set<String> getQueues() 
          • Modified CapacityTaskScheduler to check the number of queues configured in capacity_scheduler.xml against queues in QueueManager
          • Added a test case to test new method in CapacitySchedulerConf
          • Modified TestCapacityScheduler to support getQueues()
          Show
          Sreekanth Ramakrishnan added a comment - Attaching patch with following changes: Added CapacitySchedulerConf Set< String > getQueues() Modified CapacityTaskScheduler to check the number of queues configured in capacity_scheduler.xml against queues in QueueManager Added a test case to test new method in CapacitySchedulerConf Modified TestCapacityScheduler to support getQueues()
          Hide
          Hemanth Yamijala added a comment -
          • The warning statement refers to hadoop-defaults, but the configuration can be in hadoop-site as well. I would suggest we print something like: "Queue <queue-name> is not configured in mapred.queue.names property in site configuration."
          • The test case testGetQueues must configure more than one queue and verify they are present. The names should also be different from 'default'.
          Show
          Hemanth Yamijala added a comment - The warning statement refers to hadoop-defaults, but the configuration can be in hadoop-site as well. I would suggest we print something like: "Queue <queue-name> is not configured in mapred.queue.names property in site configuration." The test case testGetQueues must configure more than one queue and verify they are present. The names should also be different from 'default'.
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching patch with comments from Hemanth, incorporated

          Show
          Sreekanth Ramakrishnan added a comment - Attaching patch with comments from Hemanth, incorporated
          Hide
          Hemanth Yamijala added a comment -

          Looks good to me.

          Show
          Hemanth Yamijala added a comment - Looks good to me.
          Hide
          Hemanth Yamijala added a comment -

          Running through Hudson.

          Show
          Hemanth Yamijala added a comment - Running through Hudson.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12391004/HADOOP-4079-2.patch
          against trunk revision 699676.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3389/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3389/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3389/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3389/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12391004/HADOOP-4079-2.patch against trunk revision 699676. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3389/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3389/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3389/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3389/console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          This patch doesn't apply to trunk any more.

          It doesn't seem to be a blocker for 0.19, so I'm moving it to 0.20.

          Show
          Owen O'Malley added a comment - This patch doesn't apply to trunk any more. It doesn't seem to be a blocker for 0.19, so I'm moving it to 0.20.
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching patch merging with the latest trunk. The conflict was because of the changes which went into HADOOP-4178

          Show
          Sreekanth Ramakrishnan added a comment - Attaching patch merging with the latest trunk. The conflict was because of the changes which went into HADOOP-4178
          Hide
          Sreekanth Ramakrishnan added a comment -

          Running the patch thro' Hudson.

          Show
          Sreekanth Ramakrishnan added a comment - Running the patch thro' Hudson.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12391192/HADOOP-4079-3.patch
          against trunk revision 700628.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3408/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3408/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3408/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3408/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12391192/HADOOP-4079-3.patch against trunk revision 700628. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3408/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3408/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3408/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3408/console This message is automatically generated.
          Hide
          Sreekanth Ramakrishnan added a comment -

          The core test failed is not releated to this issue.

          Show
          Sreekanth Ramakrishnan added a comment - The core test failed is not releated to this issue.
          Hide
          Sreekanth Ramakrishnan added a comment -

          Resubmitting the patch for running thro' Hudson, the patch still applies against the trunk.

          Show
          Sreekanth Ramakrishnan added a comment - Resubmitting the patch for running thro' Hudson, the patch still applies against the trunk.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12391192/HADOOP-4079-3.patch
          against trunk revision 718232.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3600/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3600/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3600/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3600/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12391192/HADOOP-4079-3.patch against trunk revision 718232. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3600/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3600/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3600/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3600/console This message is automatically generated.
          Hide
          Sreekanth Ramakrishnan added a comment -

          Patch does not apply to the trunk cleanly anymore

          Show
          Sreekanth Ramakrishnan added a comment - Patch does not apply to the trunk cleanly anymore
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching latest patch merged with the trunk

          Show
          Sreekanth Ramakrishnan added a comment - Attaching latest patch merged with the trunk
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12395256/HADOOP-4079-4.patch
          against trunk revision 724531.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3691/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3691/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3691/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3691/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12395256/HADOOP-4079-4.patch against trunk revision 724531. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3691/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3691/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3691/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3691/console This message is automatically generated.
          Hide
          Sreekanth Ramakrishnan added a comment -

          The core tests failed are not related to the pacth. The patch modifies only the contrib project capacity scheduler.

          Show
          Sreekanth Ramakrishnan added a comment - The core tests failed are not related to the pacth. The patch modifies only the contrib project capacity scheduler.
          Hide
          Tom White added a comment -

          This patch does not apply anymore, so removing from queue while this is fixed.

          Show
          Tom White added a comment - This patch does not apply anymore, so removing from queue while this is fixed.
          Hide
          Sreekanth Ramakrishnan added a comment -

          This issue does not apply after the commit of MAPREDUCE-860 the queue information is no more maintained in the capacity-scheduler.xml rather the properties are maintained in the mapred-queues.xml. So closing this issue. if required for older version please feel free to re-open it.

          Show
          Sreekanth Ramakrishnan added a comment - This issue does not apply after the commit of MAPREDUCE-860 the queue information is no more maintained in the capacity-scheduler.xml rather the properties are maintained in the mapred-queues.xml . So closing this issue. if required for older version please feel free to re-open it.

            People

            • Assignee:
              Sreekanth Ramakrishnan
              Reporter:
              Hemanth Yamijala
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development