Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-336

The logging level of the tasks should be configurable by the job

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      Allow logging level of map/reduce tasks to be configurable.
      Configuration changes:
        add mapred.map.child.log.level
        add mapred.reduce.child.log.level
      Show
      Allow logging level of map/reduce tasks to be configurable. Configuration changes:   add mapred.map.child.log.level   add mapred.reduce.child.log.level

      Description

      It would be nice to be able to configure the logging level of the Task JVM's separately from the server JVM's. Reducing logging substantially increases performance and reduces the consumption of local disk on the task trackers.

        Issue Links

          Activity

          Arun C Murthy made changes -
          Link This issue is a clone of MAPREDUCE-5408 [ MAPREDUCE-5408 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Release Note Allow logging level of map/reduce tasks to be configurable.
          Configuration changes:
            add mapred.map.child.log.level
            add mapred.reduce.child.log.level
          Resolution Fixed [ 1 ]
          Hide
          Arun C Murthy added a comment -

          I just committed this.

          Show
          Arun C Murthy added a comment - I just committed this.
          Hide
          Devaraj Das added a comment -

          Patch looks fine to me.

          Show
          Devaraj Das added a comment - Patch looks fine to me.
          Hide
          Arun C Murthy added a comment -

          TestJobHistory failure is being tracked at MAPREDUCE-882.

          I've opened MAPREDUCE-891 to track streaming test-failures.

          Show
          Arun C Murthy added a comment - TestJobHistory failure is being tracked at MAPREDUCE-882 . I've opened MAPREDUCE-891 to track streaming test-failures.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12416955/MAPREDUCE-336_0_20090818.patch
          against trunk revision 805324.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416955/MAPREDUCE-336_0_20090818.patch against trunk revision 805324. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/console This message is automatically generated.
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.21.0 [ 12314045 ]
          Arun C Murthy made changes -
          Attachment MAPREDUCE-336_0_20090818.patch [ 12416955 ]
          Hide
          Arun C Murthy added a comment -

          Straight-forward fix.

          Show
          Arun C Murthy added a comment - Straight-forward fix.
          Arun C Murthy made changes -
          Assignee Arun C Murthy [ acmurthy ]
          Owen O'Malley made changes -
          Field Original Value New Value
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-5786 MAPREDUCE-336
          Component/s mapred [ 12310690 ]
          Hide
          Hong Tang added a comment -

          With regard to storage consumption of logs, we could separate debug/info logging from warn/error logging, and employ log rotation on the former. This would prevent disk exhaustion even if debug/info are not disabled.

          Show
          Hong Tang added a comment - With regard to storage consumption of logs, we could separate debug/info logging from warn/error logging, and employ log rotation on the former. This would prevent disk exhaustion even if debug/info are not disabled.
          Owen O'Malley created issue -

            People

            • Assignee:
              Arun C Murthy
              Reporter:
              Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development