Hadoop Common
  1. Hadoop Common
  2. HADOOP-5440

Successful taskid are not removed from TaskMemoryManager

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.19.1
    • Fix Version/s: 0.19.2, 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Successfully completed task-attempt-ids are not removed from TaskMemoryManager. This is after refactoring the code in tracker.reportTaskFinished into tip.reportTaskFinished, in HADOOP-4759

      1. patch-5440.txt
        0.6 kB
        Amareshwari Sriramadasu
      2. patch-5440-1.txt
        1 kB
        Amareshwari Sriramadasu

        Activity

        Hide
        Vinod Kumar Vavilapalli added a comment -

        HADOOP-4759 is into 19 and above. And the current bug results in a memory leak that'll be a big issue on long running clusters. Marking this as a blocker for 0.20; we also need this to be fixed for 0.19.

        Show
        Vinod Kumar Vavilapalli added a comment - HADOOP-4759 is into 19 and above. And the current bug results in a memory leak that'll be a big issue on long running clusters. Marking this as a blocker for 0.20; we also need this to be fixed for 0.19.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch removing the task entry from MemoryManager in reportDone() method.

        Show
        Amareshwari Sriramadasu added a comment - Patch removing the task entry from MemoryManager in reportDone() method.
        Hide
        Devaraj Das added a comment -

        Please remove the call to removeFromMemoryManager call from within the reportTaskFinished. Have the removeFromMemoryManager in only the taskFinished call since the latter gets called for both successful and failed tasks.

        Show
        Devaraj Das added a comment - Please remove the call to removeFromMemoryManager call from within the reportTaskFinished. Have the removeFromMemoryManager in only the taskFinished call since the latter gets called for both successful and failed tasks.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch incorporating Devaraj's comments.

        Show
        Amareshwari Sriramadasu added a comment - Patch incorporating Devaraj's comments.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch applies to trunk, branch 0.20 and branch 0.19

        Show
        Amareshwari Sriramadasu added a comment - Patch applies to trunk, branch 0.20 and branch 0.19
        Hide
        Amareshwari Sriramadasu added a comment -

        test-patch result:

             [exec]
             [exec] -1 overall.
             [exec]
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec]
             [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
             [exec]                         Please justify why no tests are needed for this patch.
             [exec]
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec]
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec]
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec]
             [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
             [exec]
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
             [exec]
         

        Manually tested memory-leak.
        All tests passed on my machine except TestCLI (HADOOP-5477)

        Show
        Amareshwari Sriramadasu added a comment - test-patch result: [exec] [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] Manually tested memory-leak. All tests passed on my machine except TestCLI ( HADOOP-5477 )
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12402012/patch-5440-1.txt
        against trunk revision 752779.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12402012/patch-5440-1.txt against trunk revision 752779. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/75/console This message is automatically generated.
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Amareshwari!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Amareshwari!
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #778 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/778/ )

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Amareshwari Sriramadasu
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development