INode interlinks can multiply effect of memory leaks

• Key: HADOOP-3381
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Raghu Angadi
• Reporter: Raghu Angadi
• Affects Version/s: 0.16.0
• Fix Version/s: 0.18.0

Description

Say a directory 'DIR' has a directory tree under it with 10000 files and directories. Each INode keeps refs to parent and children. When DIR is deleted, memory wise we essentially delete link from its parent (and delete the all the blocks from blocksMap). We don't modify its children. This is ok since this will form an island of references and will be gc-ed. Thats when everything is perfect. But if there is a bug that leaves a ref from a valid object (there is a suspect, I will another jira) to even one of these 10000 files, it could hold up all the INode and related objects. This can make a smaller mem leak many times more severe.

Comments

[Tsz Wo (Nicholas), SZE]

The INode.parent reference is actually not necessary and waste space. It would be good if we can remove it.

[Raghu Angadi]

> The INode.parent reference is actually not necessary and waste space. It would be good if we can remove it.

This is related but different issue. I am planning a patch that is suitable for 0.16 and up. Even I thought parent link was removed earlier (to save memory).

Note that parent is not the only culprit. the links to children also have the similar negative effects.

We already iterate over all the files below a directory when it is deleted, so clearing a few more references will be a straight fwd fix.

[Raghu Angadi]

Attached patch removes the links. Manually verified that even when we revert the fix for HADOOP-3382, only the INode that is directly affected is leaked.

Renamed collectSubtreeBlocks() to collectSubtreeBlocksAndClear() to reflect the change in functionality. Suggestion for better name is welcome.

[Lohit Vijayarenu]

+1 looks good

[Raghu Angadi]

This is not a functionality change, rather minimizing effect of memory leaks. I don't think it needs a test case.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12382217/HADOOP-3381.patch
against trunk revision 656939.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2494/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2494/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2494/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2494/console

This message is automatically generated.

[Raghu Angadi]

I just committed this.

[Hudson]

Integrated in Hadoop-trunk #499 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/499/)