Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-832

Too many WARN messages about deprecated memorty config variables in JobTacker log

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.20.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Reduced the frequency of log messages printed when a deprecated memory management variable is found in configuration of a job.

      Description

      When user submit a mapred job using old memory config vairiable (mapred.task.maxmem) followinig message too many times in JobTracker logs -:
      [
      WARN org.apache.hadoop.mapred.JobConf: The variable mapred.task.maxvmem is no longer used instead use mapred.job.map.memory.mb and mapred.job.reduce.memory.mb
      ]

      1. mapreduce-832-20.patch
        3 kB
        Hemanth Yamijala
      2. mapreduce-832.patch
        3 kB
        Hemanth Yamijala
      3. mapreduce-832-20.patch
        3 kB
        rahul k singh

        Activity

        Karam Singh created issue -
        Karam Singh made changes -
        Field Original Value New Value
        Summary Too man y WARN messages about deprecated memorty config variables in JobTacker log Too many WARN messages about deprecated memorty config variables in JobTacker log
        Hide
        rahul k singh added a comment -

        Added the new patch.
        All the warning messages are moved to constructor

        Show
        rahul k singh added a comment - Added the new patch. All the warning messages are moved to constructor
        rahul k singh made changes -
        Attachment mapreduce-832-20.patch [ 12416060 ]
        Hide
        Hemanth Yamijala added a comment -

        Attached a new patch that works for trunk. It is the same as what Rahul uploaded, except I modified the method checkAndWarnDeprecation to not require a Configuration instance. Instead, it uses the current object's values itself. Running this through Hudson

        Show
        Hemanth Yamijala added a comment - Attached a new patch that works for trunk. It is the same as what Rahul uploaded, except I modified the method checkAndWarnDeprecation to not require a Configuration instance. Instead, it uses the current object's values itself. Running this through Hudson
        Hemanth Yamijala made changes -
        Attachment mapreduce-832.patch [ 12416966 ]
        Hemanth Yamijala made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee rahul k singh [ rksingh ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12416966/mapreduce-832.patch
        against trunk revision 805324.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/493/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/493/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/493/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/493/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416966/mapreduce-832.patch against trunk revision 805324. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/493/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/493/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/493/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/493/console This message is automatically generated.
        Hide
        Hemanth Yamijala added a comment -

        -1 tests included. The patch doesn't appear to include any new or modified tests.

        The patch is adding a debug log message which is hard to write a unit test for. Our QE team has manually verified the patch by running tests on a cluster with old memory configuration and verifying that the number of WARN messages has come down drastically. Also, the scenario in MAPREDUCE-833 has been tested and found fixed.

        -1 core tests. The patch failed core unit tests.

        org.apache.hadoop.mapred.TestRecoveryManager.testRestartCount's failure is independent of this patch. It happens with trunk and is being tracked in MAPREDUCE-880

        -1 contrib tests. The patch failed contrib unit tests.

        The streaming test failures have been happening for a while on hudson alone. They pass locally.

        On this basis, I am going to commit the patch to trunk and an equivalent patch to branch 20.

        Show
        Hemanth Yamijala added a comment - -1 tests included. The patch doesn't appear to include any new or modified tests. The patch is adding a debug log message which is hard to write a unit test for. Our QE team has manually verified the patch by running tests on a cluster with old memory configuration and verifying that the number of WARN messages has come down drastically. Also, the scenario in MAPREDUCE-833 has been tested and found fixed. -1 core tests. The patch failed core unit tests. org.apache.hadoop.mapred.TestRecoveryManager.testRestartCount's failure is independent of this patch. It happens with trunk and is being tracked in MAPREDUCE-880 -1 contrib tests. The patch failed contrib unit tests. The streaming test failures have been happening for a while on hudson alone. They pass locally. On this basis, I am going to commit the patch to trunk and an equivalent patch to branch 20.
        Hide
        Hemanth Yamijala added a comment -

        Patch for Hadoop 0.20 branch. Only source paths have been changed.

        Show
        Hemanth Yamijala added a comment - Patch for Hadoop 0.20 branch. Only source paths have been changed.
        Hemanth Yamijala made changes -
        Attachment mapreduce-832-20.patch [ 12417102 ]
        Hide
        Hemanth Yamijala added a comment -

        I just committed this to trunk and branch 0.20. Thanks, Rahul !

        Show
        Hemanth Yamijala added a comment - I just committed this to trunk and branch 0.20. Thanks, Rahul !
        Hemanth Yamijala made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.20.1 [ 12314047 ]
        Resolution Fixed [ 1 ]
        Hemanth Yamijala made changes -
        Release Note Reduced the frequency of log messages printed when a deprecated memory management variable is found in configuration of a job.

          People

          • Assignee:
            rahul k singh
            Reporter:
            Karam Singh
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development