Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1160

Two log statements at INFO level fill up jobtracker logs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1, 0.20.2, 0.20.3
    • Fix Version/s: 0.21.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Changed some log statements that were filling up jobtracker logs to debug level.

      Description

      There are two log statements being logged at an INFO level that are unnecessarily filling up JT logs. This is making it difficult to debug issues on large cluster systems.

      The two statements identified are the following:

      INFO org.apache.hadoop.mapred.JobInProgress: No reduces to schedule for <jobid>
      and
      INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:22 completedMapsInputSize:656 completedMapsOutputSize:6299

      1. MAPREDUCE-1160-20.patch
        2 kB
        Hemanth Yamijala
      2. MR-1160-2.patch
        2 kB
        Hong Tang
      3. MR-1160.patch
        2 kB
        Ravi Gummadi

        Activity

        Hide
        Hemanth Yamijala added a comment -

        We can move these to debug level under an if (LOG.isDebugEnabled()) check.

        Show
        Hemanth Yamijala added a comment - We can move these to debug level under an if (LOG.isDebugEnabled()) check.
        Hide
        Ravi Gummadi added a comment -

        Attaching patch by changing the log statements to debug level.

        Show
        Ravi Gummadi added a comment - Attaching patch by changing the log statements to debug level.
        Hide
        Hemanth Yamijala added a comment -

        +1.

        Show
        Hemanth Yamijala added a comment - +1.
        Hide
        Amar Kamat added a comment -

        Ravi,
        Can you please check JobTracker, JobInProgress and TaskInProgress for similar cases?

        Show
        Amar Kamat added a comment - Ravi, Can you please check JobTracker, JobInProgress and TaskInProgress for similar cases?
        Hide
        Ravi Gummadi added a comment -

        Checked JobTracker, JobInProgress and TaskInProgress. I don't see any other similar cases there.

        Show
        Ravi Gummadi added a comment - Checked JobTracker, JobInProgress and TaskInProgress. I don't see any other similar cases there.
        Hide
        Amar Kamat added a comment -

        +1

        Show
        Amar Kamat added a comment - +1
        Hide
        Hong Tang added a comment -

        When testing mumak, we also found that a few LOG.debug() statements include unnecessary string concatenation operations. This is probably a good place to fix those too. I will upload a updated patch shortly.

        Show
        Hong Tang added a comment - When testing mumak, we also found that a few LOG.debug() statements include unnecessary string concatenation operations. This is probably a good place to fix those too. I will upload a updated patch shortly.
        Hide
        Hong Tang added a comment -

        Revision to my previous statement:

        When testing mumak, we found 24% of the cpu time is spent in string concatenation operations in the LOG.debug() statement in ResourceEstimator.getEstimatedTotalMapOutputSize()

        Show
        Hong Tang added a comment - Revision to my previous statement: When testing mumak, we found 24% of the cpu time is spent in string concatenation operations in the LOG.debug() statement in ResourceEstimator.getEstimatedTotalMapOutputSize()
        Hide
        Hemanth Yamijala added a comment -

        Canceling patch to take care of Hong's new patch.

        Show
        Hemanth Yamijala added a comment - Canceling patch to take care of Hong's new patch.
        Hide
        Hemanth Yamijala added a comment -

        +1 for Hong's patch. Running through Hudson.

        Show
        Hemanth Yamijala added a comment - +1 for Hong's patch. Running through Hudson.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12423427/MR-1160.patch
        against trunk revision 830531.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/98/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/98/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/98/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/98/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423427/MR-1160.patch against trunk revision 830531. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/98/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/98/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/98/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/98/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12423467/MR-1160-2.patch
        against trunk revision 830531.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/100/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/100/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/100/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/100/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423467/MR-1160-2.patch against trunk revision 830531. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/100/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/100/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/100/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/100/console This message is automatically generated.
        Hide
        Hemanth Yamijala added a comment -

        The -1 from tests is expected because this is only a change to log lines and hence no tests were included. I will commit this patch now.

        It is surprising though that we are getting a +1 overall while tests failed. Clearly something to fix urgently. I will open a separate JIRA for that.

        Show
        Hemanth Yamijala added a comment - The -1 from tests is expected because this is only a change to log lines and hence no tests were included. I will commit this patch now. It is surprising though that we are getting a +1 overall while tests failed. Clearly something to fix urgently. I will open a separate JIRA for that.
        Hide
        Hemanth Yamijala added a comment -

        It is surprising though that we are getting a +1 overall while tests failed. Clearly something to fix urgently. I will open a separate JIRA for that.

        HADOOP-6341.

        Show
        Hemanth Yamijala added a comment - It is surprising though that we are getting a +1 overall while tests failed. Clearly something to fix urgently. I will open a separate JIRA for that. HADOOP-6341 .
        Hide
        Hemanth Yamijala added a comment -

        I committed this to trunk. Thanks, Ravi and Hong !

        Show
        Hemanth Yamijala added a comment - I committed this to trunk. Thanks, Ravi and Hong !
        Hide
        Hemanth Yamijala added a comment -

        Attaching a patch for Hadoop 0.20. (Not for commit).

        Show
        Hemanth Yamijala added a comment - Attaching a patch for Hadoop 0.20. (Not for commit).
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #101 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/101/)
        . Reduce verbosity of log lines in some Map/Reduce classes to avoid filling up jobtracker logs on a busy cluster. Contributed by Ravi Gummadi and Hong Tang.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #101 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/101/ ) . Reduce verbosity of log lines in some Map/Reduce classes to avoid filling up jobtracker logs on a busy cluster. Contributed by Ravi Gummadi and Hong Tang.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #127 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/127/)
        . Reduce verbosity of log lines in some Map/Reduce classes to avoid filling up jobtracker logs on a busy cluster. Contributed by Ravi Gummadi and Hong Tang.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #127 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/127/ ) . Reduce verbosity of log lines in some Map/Reduce classes to avoid filling up jobtracker logs on a busy cluster. Contributed by Ravi Gummadi and Hong Tang.
        Hide
        gary murry added a comment -

        I agree that this does not need unit testing, because we are just changing the log level of the items, but we do need to test have tests for log entries when we can. We rely on logging to help debug issues in the wild and need to have assurance that they are working correctly. Thanks.

        Show
        gary murry added a comment - I agree that this does not need unit testing, because we are just changing the log level of the items, but we do need to test have tests for log entries when we can. We rely on logging to help debug issues in the wild and need to have assurance that they are working correctly. Thanks.
        Hide
        Allen Wittenauer added a comment -

        Let's get this fixed in the 0.20 branch too please.

        Show
        Allen Wittenauer added a comment - Let's get this fixed in the 0.20 branch too please.

          People

          • Assignee:
            Ravi Gummadi
            Reporter:
            Hemanth Yamijala
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development