Lease paths are sometimes not removed from LeaseManager.sortedLeasesByPath

• Key: HADOOP-3375
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Blocker
• Assignee: Tsz Wo (Nicholas), SZE
• Reporter: Tsz Wo (Nicholas), SZE
• Fix Version/s: 0.18.0

Description

In LeaseManager,

• removeExpiredLease(Lease) does not remove paths from sortedLeasesByPath
• removeLease(...) should do sortedLeasesByPath.remove(src) no matter lease.hasPath() returns true or false.

Comments

[Tsz Wo (Nicholas), SZE]

Also, lease related structure should store names and paths as String instead of StringBytesWritable. It will prevent unnecessary converting between String and StringBytesWritable.

[Tsz Wo (Nicholas), SZE]

3375_20080512.patch: fixed sortedLeasesByPath and removed StringBytesWritable.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381917/3375_20080512.patch
against trunk revision 655674.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2454/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2454/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2454/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2454/console

This message is automatically generated.

[Robert Chansler]

Nicholas reports that this may result in the loss of a lease record for every task that creates a file. This is a new problem not present in 0.16.

[Robert Chansler]

Not a 0.17 problem after all (only trunk). I'm sorry for the confusion.

[dhruba borthakur]

+1 code looks good. Is there a way to enhance an existing unit test to catch this situation?

[Tsz Wo (Nicholas), SZE]

3375_20080514.patch: added a test

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12382068/3375_20080514.patch
against trunk revision 656270.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 4 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2471/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2471/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2471/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2471/console

This message is automatically generated.

[Tsz Wo (Nicholas), SZE]

3375_20080519.patch: reverted the change of removing StringBytesWritable since it needs more thought.

[dhruba borthakur]

I like the latest patch better since it does not affect the disk format. Changing the StringBytesWritbale is bets done in a separate patch.

[Tsz Wo (Nicholas), SZE]

> Changing the StringBytesWritbale is bets done in a separate patch.

Yes, I plan to create another issue for removing StringBytesWritbale which was introduced in HADOOP-1283.

[Tsz Wo (Nicholas), SZE]

3375_20080520.patch: updated javadoc

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12382400/3375_20080520.patch
against trunk revision 658035.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 4 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2504/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2504/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2504/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2504/console

This message is automatically generated.

[Tsz Wo (Nicholas), SZE]

Two tests failed but none of them is related to the patch.

• See HADOOP-3419 for TestFsck

• See HADOOP-3354 for TestEditLog

[dhruba borthakur]

+1. Code looks good.

[dhruba borthakur]

I just committed this. Thanks Nicholas!

[Hudson]

Integrated in Hadoop-trunk #499 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/499/)

[Tsz Wo (Nicholas), SZE]

> Yes, I plan to create another issue for removing StringBytesWritbale which was introduced in HADOOP-1283.

Created HADOOP-3461