Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-336

The logging level of the tasks should be configurable by the job

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      Allow logging level of map/reduce tasks to be configurable.
      Configuration changes:
        add mapred.map.child.log.level
        add mapred.reduce.child.log.level
      Show
      Allow logging level of map/reduce tasks to be configurable. Configuration changes:   add mapred.map.child.log.level   add mapred.reduce.child.log.level

      Description

      It would be nice to be able to configure the logging level of the Task JVM's separately from the server JVM's. Reducing logging substantially increases performance and reduces the consumption of local disk on the task trackers.

        Issue Links

          Activity

          Hide
          Hong Tang added a comment -

          With regard to storage consumption of logs, we could separate debug/info logging from warn/error logging, and employ log rotation on the former. This would prevent disk exhaustion even if debug/info are not disabled.

          Show
          Hong Tang added a comment - With regard to storage consumption of logs, we could separate debug/info logging from warn/error logging, and employ log rotation on the former. This would prevent disk exhaustion even if debug/info are not disabled.
          Hide
          Arun C Murthy added a comment -

          Straight-forward fix.

          Show
          Arun C Murthy added a comment - Straight-forward fix.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12416955/MAPREDUCE-336_0_20090818.patch
          against trunk revision 805324.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416955/MAPREDUCE-336_0_20090818.patch against trunk revision 805324. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/492/console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          TestJobHistory failure is being tracked at MAPREDUCE-882.

          I've opened MAPREDUCE-891 to track streaming test-failures.

          Show
          Arun C Murthy added a comment - TestJobHistory failure is being tracked at MAPREDUCE-882 . I've opened MAPREDUCE-891 to track streaming test-failures.
          Hide
          Devaraj Das added a comment -

          Patch looks fine to me.

          Show
          Devaraj Das added a comment - Patch looks fine to me.
          Hide
          Arun C Murthy added a comment -

          I just committed this.

          Show
          Arun C Murthy added a comment - I just committed this.

            People

            • Assignee:
              Arun C Murthy
              Reporter:
              Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development