Hadoop Common
  1. Hadoop Common
  2. HADOOP-3375

Lease paths are sometimes not removed from LeaseManager.sortedLeasesByPath

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In LeaseManager,

      • removeExpiredLease(Lease) does not remove paths from sortedLeasesByPath
      • removeLease(...) should do sortedLeasesByPath.remove(src) no matter lease.hasPath() returns true or false.
      1. 3375_20080520.patch
        5 kB
        Tsz Wo Nicholas Sze
      2. 3375_20080519.patch
        5 kB
        Tsz Wo Nicholas Sze
      3. 3375_20080514.patch
        15 kB
        Tsz Wo Nicholas Sze
      4. 3375_20080512.patch
        12 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Yes, I plan to create another issue for removing StringBytesWritbale which was introduced in HADOOP-1283.

          Created HADOOP-3461

          Show
          Tsz Wo Nicholas Sze added a comment - > Yes, I plan to create another issue for removing StringBytesWritbale which was introduced in HADOOP-1283 . Created HADOOP-3461
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #499 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/499/ )
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Nicholas!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Nicholas!
          Hide
          dhruba borthakur added a comment -

          +1. Code looks good.

          Show
          dhruba borthakur added a comment - +1. Code looks good.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Two tests failed but none of them is related to the patch.

          Show
          Tsz Wo Nicholas Sze added a comment - Two tests failed but none of them is related to the patch. See HADOOP-3419 for TestFsck See HADOOP-3354 for TestEditLog
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12382400/3375_20080520.patch
          against trunk revision 658035.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2504/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2504/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2504/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2504/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12382400/3375_20080520.patch against trunk revision 658035. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2504/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2504/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2504/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2504/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          3375_20080520.patch: updated javadoc

          Show
          Tsz Wo Nicholas Sze added a comment - 3375_20080520.patch: updated javadoc
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Changing the StringBytesWritbale is bets done in a separate patch.

          Yes, I plan to create another issue for removing StringBytesWritbale which was introduced in HADOOP-1283.

          Show
          Tsz Wo Nicholas Sze added a comment - > Changing the StringBytesWritbale is bets done in a separate patch. Yes, I plan to create another issue for removing StringBytesWritbale which was introduced in HADOOP-1283 .
          Hide
          dhruba borthakur added a comment -

          I like the latest patch better since it does not affect the disk format. Changing the StringBytesWritbale is bets done in a separate patch.

          Show
          dhruba borthakur added a comment - I like the latest patch better since it does not affect the disk format. Changing the StringBytesWritbale is bets done in a separate patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          3375_20080519.patch: reverted the change of removing StringBytesWritable since it needs more thought.

          Show
          Tsz Wo Nicholas Sze added a comment - 3375_20080519.patch: reverted the change of removing StringBytesWritable since it needs more thought.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12382068/3375_20080514.patch
          against trunk revision 656270.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2471/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2471/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2471/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2471/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12382068/3375_20080514.patch against trunk revision 656270. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2471/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2471/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2471/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2471/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          3375_20080514.patch: added a test

          Show
          Tsz Wo Nicholas Sze added a comment - 3375_20080514.patch: added a test
          Hide
          dhruba borthakur added a comment -

          +1 code looks good. Is there a way to enhance an existing unit test to catch this situation?

          Show
          dhruba borthakur added a comment - +1 code looks good. Is there a way to enhance an existing unit test to catch this situation?
          Hide
          Robert Chansler added a comment -

          Not a 0.17 problem after all (only trunk). I'm sorry for the confusion.

          Show
          Robert Chansler added a comment - Not a 0.17 problem after all (only trunk). I'm sorry for the confusion.
          Hide
          Robert Chansler added a comment -

          Nicholas reports that this may result in the loss of a lease record for every task that creates a file. This is a new problem not present in 0.16.

          Show
          Robert Chansler added a comment - Nicholas reports that this may result in the loss of a lease record for every task that creates a file. This is a new problem not present in 0.16.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12381917/3375_20080512.patch
          against trunk revision 655674.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2454/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2454/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2454/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2454/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12381917/3375_20080512.patch against trunk revision 655674. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2454/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2454/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2454/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2454/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          3375_20080512.patch: fixed sortedLeasesByPath and removed StringBytesWritable.

          Show
          Tsz Wo Nicholas Sze added a comment - 3375_20080512.patch: fixed sortedLeasesByPath and removed StringBytesWritable.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Also, lease related structure should store names and paths as String instead of StringBytesWritable. It will prevent unnecessary converting between String and StringBytesWritable.

          Show
          Tsz Wo Nicholas Sze added a comment - Also, lease related structure should store names and paths as String instead of StringBytesWritable. It will prevent unnecessary converting between String and StringBytesWritable.

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development