Hadoop Common
  1. Hadoop Common
  2. HADOOP-5440

Successful taskid are not removed from TaskMemoryManager

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.19.1
    • Fix Version/s: 0.19.2, 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Successfully completed task-attempt-ids are not removed from TaskMemoryManager. This is after refactoring the code in tracker.reportTaskFinished into tip.reportTaskFinished, in HADOOP-4759

      1. patch-5440.txt
        0.6 kB
        Amareshwari Sriramadasu
      2. patch-5440-1.txt
        1 kB
        Amareshwari Sriramadasu

        Activity

        Amareshwari Sriramadasu created issue -
        Amareshwari Sriramadasu made changes -
        Field Original Value New Value
        Assignee Amareshwari Sriramadasu [ amareshwari ]
        Hide
        Vinod Kumar Vavilapalli added a comment -

        HADOOP-4759 is into 19 and above. And the current bug results in a memory leak that'll be a big issue on long running clusters. Marking this as a blocker for 0.20; we also need this to be fixed for 0.19.

        Show
        Vinod Kumar Vavilapalli added a comment - HADOOP-4759 is into 19 and above. And the current bug results in a memory leak that'll be a big issue on long running clusters. Marking this as a blocker for 0.20; we also need this to be fixed for 0.19.
        Vinod Kumar Vavilapalli made changes -
        Priority Major [ 3 ] Blocker [ 1 ]
        Vinod Kumar Vavilapalli made changes -
        Fix Version/s 0.20.0 [ 12313438 ]
        Fix Version/s 0.21.0 [ 12313563 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch removing the task entry from MemoryManager in reportDone() method.

        Show
        Amareshwari Sriramadasu added a comment - Patch removing the task entry from MemoryManager in reportDone() method.
        Amareshwari Sriramadasu made changes -
        Attachment patch-5440.txt [ 12401903 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Devaraj Das added a comment -

        Please remove the call to removeFromMemoryManager call from within the reportTaskFinished. Have the removeFromMemoryManager in only the taskFinished call since the latter gets called for both successful and failed tasks.

        Show
        Devaraj Das added a comment - Please remove the call to removeFromMemoryManager call from within the reportTaskFinished. Have the removeFromMemoryManager in only the taskFinished call since the latter gets called for both successful and failed tasks.
        Devaraj Das made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch incorporating Devaraj's comments.

        Show
        Amareshwari Sriramadasu added a comment - Patch incorporating Devaraj's comments.
        Amareshwari Sriramadasu made changes -
        Attachment patch-5440-1.txt [ 12402012 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch applies to trunk, branch 0.20 and branch 0.19

        Show
        Amareshwari Sriramadasu added a comment - Patch applies to trunk, branch 0.20 and branch 0.19
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s 0.19.2 [ 12313650 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        test-patch result:

             [exec]
             [exec] -1 overall.
             [exec]
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec]
             [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
             [exec]                         Please justify why no tests are needed for this patch.
             [exec]
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec]
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec]
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec]
             [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
             [exec]
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
             [exec]
         

        Manually tested memory-leak.
        All tests passed on my machine except TestCLI (HADOOP-5477)

        Show
        Amareshwari Sriramadasu added a comment - test-patch result: [exec] [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] Manually tested memory-leak. All tests passed on my machine except TestCLI ( HADOOP-5477 )
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12402012/patch-5440-1.txt
        against trunk revision 752779.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12402012/patch-5440-1.txt against trunk revision 752779. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/console This message is automatically generated.
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Amareshwari!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Amareshwari!
        Devaraj Das made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #778 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/778/ )
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s mapred [ 12310690 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        3h 47m 1 Devaraj Das 11/Mar/09 11:58
        Open Open Patch Available Patch Available
        2d 16h 51m 2 Amareshwari Sriramadasu 12/Mar/09 04:28
        Patch Available Patch Available Resolved Resolved
        7h 33m 1 Devaraj Das 12/Mar/09 12:01
        Resolved Resolved Closed Closed
        42d 7h 16m 1 Nigel Daley 23/Apr/09 20:18

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Amareshwari Sriramadasu
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development