Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1090

Modify log statement in Tasktracker log related to memory monitoring to include attempt id.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.21.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Modified log statement in task memory monitoring thread to include task attempt id.

      Description

      Currently the TaskMemoryManagerThread logs a line like:
      org.apache.hadoop.mapred.TaskMemoryManagerThread: Memory usage of ProcessTree 14321 :372686848bytes. Limit : 2147483648bytes.
      It would be very useful to include the Task attempt id for the process tree mentioned in the log statement.

      1. MAPREDUCE-1090.patch
        1 kB
        Hemanth Yamijala
      2. MAPREDUCE-1090-20.patch
        1 kB
        Hemanth Yamijala

        Activity

        Hide
        Hemanth Yamijala added a comment -

        The attached patch modifies the log line being printed in the TaskMemoryManagerThread as follows:

        Memory usage of ProcessTree <pid> for task-id <tid> : <current-usage> bytes, limit : <limit> bytes

        For e.g.

        Memory usage of ProcessTree 20889 for task-id attempt_200910221038_0001_r_000000_0 : 371458048 bytes, limit : 2147483648 bytes

        Show
        Hemanth Yamijala added a comment - The attached patch modifies the log line being printed in the TaskMemoryManagerThread as follows: Memory usage of ProcessTree <pid> for task-id <tid> : <current-usage> bytes, limit : <limit> bytes For e.g. Memory usage of ProcessTree 20889 for task-id attempt_200910221038_0001_r_000000_0 : 371458048 bytes, limit : 2147483648 bytes
        Hide
        Hemanth Yamijala added a comment -

        Since this is a log line, I have not included test cases in the patch. But I manually tested this patch with the following tests:

        • Ran a sleep job with a huge sleep time. While the job ran, looked at the tasktracker logs to see that the values were coming correctly.
        • Ran a sleep job with a modified limit. While the job ran, verified the changed limit parameter was printed in the logs.

        Can someone please review ?

        Show
        Hemanth Yamijala added a comment - Since this is a log line, I have not included test cases in the patch. But I manually tested this patch with the following tests: Ran a sleep job with a huge sleep time. While the job ran, looked at the tasktracker logs to see that the values were coming correctly. Ran a sleep job with a modified limit. While the job ran, verified the changed limit parameter was printed in the logs. Can someone please review ?
        Hide
        Arun C Murthy added a comment -

        +1

        Show
        Arun C Murthy added a comment - +1
        Hide
        Hemanth Yamijala added a comment -

        Trying hudson. Expecting a -1 on tests for reasons I explained.

        Show
        Hemanth Yamijala added a comment - Trying hudson. Expecting a -1 on tests for reasons I explained.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12422873/MAPREDUCE-1090.patch
        against trunk revision 828733.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/201/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/201/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/201/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/201/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422873/MAPREDUCE-1090.patch against trunk revision 828733. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/201/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/201/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/201/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/201/console This message is automatically generated.
        Hide
        Hemanth Yamijala added a comment -

        Since the absence of test cases is explained, I committed this to trunk.

        Show
        Hemanth Yamijala added a comment - Since the absence of test cases is explained, I committed this to trunk.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #99 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/99/)
        . Modified log statement in TaskMemoryManagerThread to include task attempt id. Contributed by Hemanth Yamijala.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #99 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/99/ ) . Modified log statement in TaskMemoryManagerThread to include task attempt id. Contributed by Hemanth Yamijala.
        Hide
        Hemanth Yamijala added a comment -

        Attached patch is for Hadoop 0.20 (not for committing).

        Show
        Hemanth Yamijala added a comment - Attached patch is for Hadoop 0.20 (not for committing).
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #127 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/127/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #127 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/127/ )

          People

          • Assignee:
            Hemanth Yamijala
            Reporter:
            Hemanth Yamijala
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development