Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.1
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None

      Description

      Huge number of the same WARN messages are written. We only need to write each distinct message once. The messages are of the form:

      2012-04-05 03:55:04,166 WARN mapreduce.Counters: Group {oldGroup} is deprecated. Use {newGroup} instead
      
      1. MAPREDUCE-4129_rev2.patch
        2 kB
        Ahmed Radwan
      2. MAPREDUCE-4129.patch
        1 kB
        Ahmed Radwan

        Activity

        Hide
        Ahmed Radwan added a comment -

        Attaching a patch addressing this issue. The patch modifies the behavior to only write the log message the first time the oldGroupName is found, instead of every time getGroup(..) is called.
        I don't think we need a test case for this change as it is a straightforward change in message logging.

        Show
        Ahmed Radwan added a comment - Attaching a patch addressing this issue. The patch modifies the behavior to only write the log message the first time the oldGroupName is found, instead of every time getGroup(..) is called. I don't think we need a test case for this change as it is a straightforward change in message logging.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12522044/MAPREDUCE-4129.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
        org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
        org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
        org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2179//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2179//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522044/MAPREDUCE-4129.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2179//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2179//console This message is automatically generated.
        Hide
        Ahmed Radwan added a comment -

        The above failures reported by jenkins don't seem to be related to this patch.

        Show
        Ahmed Radwan added a comment - The above failures reported by jenkins don't seem to be related to this patch.
        Hide
        Tom White added a comment -

        In your patch, the warning is printed every time a new group is created, even if the old and new names are the same. Don't you need to check that they are different?

        Show
        Tom White added a comment - In your patch, the warning is printed every time a new group is created, even if the old and new names are the same. Don't you need to check that they are different?
        Hide
        Ahmed Radwan added a comment -

        Thanks Tom for reviewing this! You are right. Also checking they are different doesn't seem to be sufficient, because the name could have been cut because of the length limits. I have updated the patch to check if the name was found in the legacyMap before printing the log message. Please let me know if you have further comments.

        Show
        Ahmed Radwan added a comment - Thanks Tom for reviewing this! You are right. Also checking they are different doesn't seem to be sufficient, because the name could have been cut because of the length limits. I have updated the patch to check if the name was found in the legacyMap before printing the log message. Please let me know if you have further comments.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12522197/MAPREDUCE-4129_rev2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
        org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
        org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
        org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2195//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2195//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522197/MAPREDUCE-4129_rev2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2195//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2195//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        Thanks Ahmed, +1 the patch looks good. I don't think we really need any new tests because we are just changing log messages. I put this into trunk, branch-2, and branch-0.23.

        Show
        Robert Joseph Evans added a comment - Thanks Ahmed, +1 the patch looks good. I don't think we really need any new tests because we are just changing log messages. I put this into trunk, branch-2, and branch-0.23.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2184 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2184/)
        MAPREDUCE-4129. Lots of unneeded counters log messages (Ahmed Radwan via bobby) (Revision 1328106)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328106
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2184 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2184/ ) MAPREDUCE-4129 . Lots of unneeded counters log messages (Ahmed Radwan via bobby) (Revision 1328106) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328106 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2110 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2110/)
        MAPREDUCE-4129. Lots of unneeded counters log messages (Ahmed Radwan via bobby) (Revision 1328106)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328106
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2110 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2110/ ) MAPREDUCE-4129 . Lots of unneeded counters log messages (Ahmed Radwan via bobby) (Revision 1328106) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328106 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2127 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2127/)
        MAPREDUCE-4129. Lots of unneeded counters log messages (Ahmed Radwan via bobby) (Revision 1328106)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328106
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2127 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2127/ ) MAPREDUCE-4129 . Lots of unneeded counters log messages (Ahmed Radwan via bobby) (Revision 1328106) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328106 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #233 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/233/)
        svn merge -c 1328106, FIXES: MAPREDUCE-4129. Lots of unneeded counters log messages (Ahmed Radwan via bobby) (Revision 1328108)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328108
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #233 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/233/ ) svn merge -c 1328106, FIXES: MAPREDUCE-4129 . Lots of unneeded counters log messages (Ahmed Radwan via bobby) (Revision 1328108) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328108 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1020 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1020/)
        MAPREDUCE-4129. Lots of unneeded counters log messages (Ahmed Radwan via bobby) (Revision 1328106)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328106
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1020 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1020/ ) MAPREDUCE-4129 . Lots of unneeded counters log messages (Ahmed Radwan via bobby) (Revision 1328106) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328106 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1055 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1055/)
        MAPREDUCE-4129. Lots of unneeded counters log messages (Ahmed Radwan via bobby) (Revision 1328106)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328106
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1055 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1055/ ) MAPREDUCE-4129 . Lots of unneeded counters log messages (Ahmed Radwan via bobby) (Revision 1328106) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328106 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java

          People

          • Assignee:
            Ahmed Radwan
            Reporter:
            Ahmed Radwan
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development