Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1527

QueueManager should issue warning if mapred-queues.xml is skipped.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      MAPREDUCE-861 added support for hierarchical queues and used a new file mapred-queues.xml to configure queues. But the QueueManager will silently skip the file if property "mapred.queue.names" is defined in Configuration. This tripped us when we are testing MAPREDUCE-1235 and copied configuration files from a Hadoop 0.20 cluster.

      I suggest QueueManager issue a friendly warning if both of the following are true: (1) "mapred.queue.names" exists in Configuration; (2) "mapred-queues.xml" is found in the classpath.

      1. mr-1527-20100224.patch
        1 kB
        Hong Tang
      2. mr-1527-20100224-2.patch
        1 kB
        Hong Tang

        Activity

        Hide
        Hong Tang added a comment -

        I mistyped in the description, "MAPREDUCE-1235" should be "MAPREDUCE-1253".

        Show
        Hong Tang added a comment - I mistyped in the description, " MAPREDUCE-1235 " should be " MAPREDUCE-1253 ".
        Hide
        Hemanth Yamijala added a comment -

        Hong, I see this in code logged as a WARNING message: (edited for showing the intent) Configuring "mapred.queue.names" in mapred-site.xml or hadoop-site.xml is deprecated. Configure queue hierarchy in QUEUE_CONF_FILE_NAME.

        This is the standard used in Hadoop for warning against deprecated keys. Does this not help enough ?

        Show
        Hemanth Yamijala added a comment - Hong, I see this in code logged as a WARNING message: (edited for showing the intent) Configuring "mapred.queue.names" in mapred-site.xml or hadoop-site.xml is deprecated. Configure queue hierarchy in QUEUE_CONF_FILE_NAME. This is the standard used in Hadoop for warning against deprecated keys. Does this not help enough ?
        Hide
        Hong Tang added a comment -

        I rerun the test and now I noticed the following message:

        "10/02/23 22:23:05 WARN mapred.DeprecatedQueueConfigurationParser: Configuring "mapred.queue.names" in mapred-site.xml or hadoop-site.xml is deprecated. Configure queue hierarchy in mapred-queues.xml".

        Looking back to reconstruct our thoughts - I believe we probably ignored this message, which is among the many similar warnings about deprecated keys in the conf file. Since we do put a properly formatted mapred-queues.xml in the classpath, we thought mapred-queues.xml should be loaded. I think the warning message could be more explicit, e.g. "Confiure queue hierarchy in mapred-queues.xml and remove "mapred.queue.names" from mapred-site.xml". Or "mapred-queues.xml will be ignored since you have "mapred.queue.names" defined in mapred-site.xml".

        Show
        Hong Tang added a comment - I rerun the test and now I noticed the following message: "10/02/23 22:23:05 WARN mapred.DeprecatedQueueConfigurationParser: Configuring "mapred.queue.names" in mapred-site.xml or hadoop-site.xml is deprecated. Configure queue hierarchy in mapred-queues.xml". Looking back to reconstruct our thoughts - I believe we probably ignored this message, which is among the many similar warnings about deprecated keys in the conf file. Since we do put a properly formatted mapred-queues.xml in the classpath, we thought mapred-queues.xml should be loaded. I think the warning message could be more explicit, e.g. "Confiure queue hierarchy in mapred-queues.xml and remove "mapred.queue.names" from mapred-site.xml". Or "mapred-queues.xml will be ignored since you have "mapred.queue.names" defined in mapred-site.xml".
        Hide
        Hong Tang added a comment -

        Trivial patch that make the message a bit more explicit:

        10/02/24 10:10:09 WARN mapred.DeprecatedQueueConfigurationParser: Configuring "mapred.queue.names" in mapred-site.xml or hadoop-site.xml is deprecated and will overshadow mapred-queues.xml. Removing this property and configure queue hierarchy in mapred-queues.xml

        Show
        Hong Tang added a comment - Trivial patch that make the message a bit more explicit: 10/02/24 10:10:09 WARN mapred.DeprecatedQueueConfigurationParser: Configuring "mapred.queue.names" in mapred-site.xml or hadoop-site.xml is deprecated and will overshadow mapred-queues.xml. Removing this property and configure queue hierarchy in mapred-queues.xml
        Hide
        Hemanth Yamijala added a comment -

        Seems OK. Minor editorial comment: I suppose the message should read "Remove this property and configure..." rather than "Removing ..."

        Show
        Hemanth Yamijala added a comment - Seems OK. Minor editorial comment: I suppose the message should read "Remove this property and configure..." rather than "Removing ..."
        Hide
        Hong Tang added a comment -

        Incorporated comments from Hemanth.

        Show
        Hong Tang added a comment - Incorporated comments from Hemanth.
        Hide
        Hong Tang added a comment -

        Trivial patch that only changes the log message. No unit test included.

        Show
        Hong Tang added a comment - Trivial patch that only changes the log message. No unit test included.
        Hide
        Hemanth Yamijala added a comment -

        +1.

        Show
        Hemanth Yamijala added a comment - +1.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12436883/mr-1527-20100224-2.patch
        against trunk revision 915223.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/481/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/481/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/481/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/481/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436883/mr-1527-20100224-2.patch against trunk revision 915223. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/481/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/481/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/481/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/481/console This message is automatically generated.
        Hide
        Arun C Murthy added a comment -

        The test failure is unrelated, this is a trivial change to a LOG.warn.

        I just committed this. Thanks Hong!

        Show
        Arun C Murthy added a comment - The test failure is unrelated, this is a trivial change to a LOG.warn. I just committed this. Thanks Hong!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #263 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/263/)
        . Better warning logged when mapred.queue.names is overshadowed by mapred-queues.xml. Contributed by Hong Tang.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #263 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/263/ ) . Better warning logged when mapred.queue.names is overshadowed by mapred-queues.xml. Contributed by Hong Tang.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #250 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/250/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #250 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/250/ )

          People

          • Assignee:
            Hong Tang
            Reporter:
            Hong Tang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development