Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1948

Forward port 'hdfs-1520 lightweight namenode operation to trigger lease reccovery'

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Adds method to NameNode/ClientProtocol that allows for rude revoke of lease on current lease holder

      Description

      This issue is about forward porting from branch-0.20-append the little namenode api that facilitates stealing of a file's lease. The forward port would be an adaption of hdfs-1520 and its companion patches, hdfs-1555 and hdfs-1554, to suit the TRUNK.

      Intent is to get this fix into 0.22 time willing; i'll run a vote to get ok on getting it added to branch. HBase needs this facility.

      1. 1948-part1.txt
        9 kB
        stack
      2. 1948-v2.txt
        18 kB
        stack
      3. 1948-v3.txt
        1 kB
        stack
      4. 1948-v3.txt
        18 kB
        stack
      5. 1948-v4.txt
        18 kB
        stack
      6. 1948-v4.22.txt
        18 kB
        stack
      7. 1948-v4-minus_rpc_version_change.txt
        18 kB
        stack

        Issue Links

          Activity

          stack created issue -
          Hide
          stack added a comment -

          Here is bulk of the port from branch-0.20-append of the recoverLease addition added by the following JIRAs.

          HDFS-1520. Lightweight NameNode operation recoverLease to trigger lease recovery. (Hairong Kuang via dhruba)
          HDFS-1555. Disallow pipelien recovery if a file is already being ease recovered (hairong)
          HDFS-1554. New semantics for recoverLease (hairong)

          Its not complete yet. The NameNode no longer has methods nextGenerationStamp (DatanodeProtocol) – they are present in 0.20.x but not in TRUNK – nor
          nextGenerationStampForBlock. I'm trying to figure what the equivalents are (Pointers welcome – thanks).

          Show
          stack added a comment - Here is bulk of the port from branch-0.20-append of the recoverLease addition added by the following JIRAs. HDFS-1520 . Lightweight NameNode operation recoverLease to trigger lease recovery. (Hairong Kuang via dhruba) HDFS-1555 . Disallow pipelien recovery if a file is already being ease recovered (hairong) HDFS-1554 . New semantics for recoverLease (hairong) Its not complete yet. The NameNode no longer has methods nextGenerationStamp (DatanodeProtocol) – they are present in 0.20.x but not in TRUNK – nor nextGenerationStampForBlock. I'm trying to figure what the equivalents are (Pointers welcome – thanks).
          stack made changes -
          Field Original Value New Value
          Attachment 1948-part1.txt [ 12480006 ]
          Hide
          stack added a comment -

          Here is full patch. Wrangling my test trying to get it to pass but putting this up meantime.

          Show
          stack added a comment - Here is full patch. Wrangling my test trying to get it to pass but putting this up meantime.
          stack made changes -
          Attachment 1948-v2.txt [ 12480607 ]
          Hide
          stack added a comment -

          Tests pass now (Had to figure the new-fangled user faking).

          Show
          stack added a comment - Tests pass now (Had to figure the new-fangled user faking).
          stack made changes -
          Attachment 1948-v3.txt [ 12480624 ]
          stack made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Adds method to NameNode/ClientProtocol that allows for rude revoke of lease on current lease holder
          Assignee stack [ stack ]
          stack made changes -
          Attachment 1948-v3.txt [ 12480625 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480625/1948-v3.txt
          against trunk revision 1128009.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/640//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480625/1948-v3.txt against trunk revision 1128009. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/640//console This message is automatically generated.
          Hide
          stack added a comment -

          Forgot to --no-prefix making patch

          Show
          stack added a comment - Forgot to --no-prefix making patch
          stack made changes -
          Attachment 1948-v4.txt [ 12480626 ]
          stack made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          stack made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480626/1948-v4.txt
          against trunk revision 1128009.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/641//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/641//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/641//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480626/1948-v4.txt against trunk revision 1128009. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/641//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/641//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/641//console This message is automatically generated.
          Hide
          stack added a comment -

          Here is a version for branch 0.22. Two minor changes only – the rpc version and how to get to leases.

          Show
          stack added a comment - Here is a version for branch 0.22. Two minor changes only – the rpc version and how to get to leases.
          stack made changes -
          Attachment 1948-v4.22.txt [ 12481434 ]
          Hide
          stack added a comment -

          I've been testing w/ this 0.22 patch and the lease grabbing seems to be working properly (We have explicit unit test in hbase for this – it passes – but did some cluster load killing of servers to test recovery and seems to do the right thing).

          Show
          stack added a comment - I've been testing w/ this 0.22 patch and the lease grabbing seems to be working properly (We have explicit unit test in hbase for this – it passes – but did some cluster load killing of servers to test recovery and seems to do the right thing).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481434/1948-v4.22.txt
          against trunk revision 1131264.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/707//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481434/1948-v4.22.txt against trunk revision 1131264. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/707//console This message is automatically generated.
          Hide
          Hairong Kuang added a comment -

          Both patches look good! Stack, thanks for your time porting this feature.

          For the trunk, we already moved away from the version based RPC compatibility. Instead we take the method name base approach. Check HADOOP-6904. So there is no need to bump the version number. Instead dfsclient should use the method name to see if the server supports recoverLease or not.

          Show
          Hairong Kuang added a comment - Both patches look good! Stack, thanks for your time porting this feature. For the trunk, we already moved away from the version based RPC compatibility. Instead we take the method name base approach. Check HADOOP-6904 . So there is no need to bump the version number. Instead dfsclient should use the method name to see if the server supports recoverLease or not.
          Hide
          stack added a comment -

          @Hairong Thanks for the review. OK with you if I commit the patches (I'll omit the bumping of rpc on commit)?

          Show
          stack added a comment - @Hairong Thanks for the review. OK with you if I commit the patches (I'll omit the bumping of rpc on commit)?
          Hide
          Hairong Kuang added a comment -

          Yes, please update the patch for the trunk.

          Show
          Hairong Kuang added a comment - Yes, please update the patch for the trunk.
          Hide
          stack added a comment -

          v4 of the patch for TRUNK minus rpc version increment.

          Show
          stack added a comment - v4 of the patch for TRUNK minus rpc version increment.
          stack made changes -
          Attachment 1948-v4-minus_rpc_version_change.txt [ 12481623 ]
          stack made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          stack made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481623/1948-v4-minus_rpc_version_change.txt
          against trunk revision 1132715.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/728//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/728//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/728//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481623/1948-v4-minus_rpc_version_change.txt against trunk revision 1132715. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/728//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/728//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/728//console This message is automatically generated.
          Hide
          stack added a comment -

          @Hairong I uploaded patch and above is what jenkins thinks of this version of the patch; its the same as v4 only minus the rpc version setting. You OK w/ my going ahead and committing this? (Thanks).

          Show
          stack added a comment - @Hairong I uploaded patch and above is what jenkins thinks of this version of the patch; its the same as v4 only minus the rpc version setting. You OK w/ my going ahead and committing this? (Thanks).
          Hide
          Hairong Kuang added a comment -

          +1 lgtm. Thanks, Stack!

          Show
          Hairong Kuang added a comment - +1 lgtm. Thanks, Stack!
          Hide
          stack added a comment -

          Applied patch to TRUNK and to branch-0.22 (HBase won't work on hadoop 0.22.0 if this patch is not present). Thanks for help and review Hairong.

          Show
          stack added a comment - Applied patch to TRUNK and to branch-0.22 (HBase won't work on hadoop 0.22.0 if this patch is not present). Thanks for help and review Hairong.
          stack made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.22.0 [ 12314241 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #66 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/66/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #66 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/66/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #746 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/746/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #746 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/746/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #699 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/699/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #699 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/699/ )
          Suresh Srinivas made changes -
          Link This issue blocks HDFS-1554 [ HDFS-1554 ]
          Hide
          Suresh Srinivas added a comment -

          HDFS-1948 forward ports HDFS-1554 to trunk

          Show
          Suresh Srinivas added a comment - HDFS-1948 forward ports HDFS-1554 to trunk
          Suresh Srinivas made changes -
          Link This issue blocks HDFS-1554 [ HDFS-1554 ]
          Suresh Srinivas made changes -
          Link This issue is related to HDFS-1554 [ HDFS-1554 ]
          Suresh Srinivas made changes -
          Link This issue is related to HDFS-1520 [ HDFS-1520 ]
          Suresh Srinivas made changes -
          Link This issue is related to HDFS-1555 [ HDFS-1555 ]
          Konstantin Shvachko made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          10d 15h 35m 2 stack 06/Jun/11 21:45
          Open Open Patch Available Patch Available
          9d 23h 38m 3 stack 06/Jun/11 21:45
          Patch Available Patch Available Resolved Resolved
          21h 1 stack 07/Jun/11 18:45
          Resolved Resolved Closed Closed
          187d 12h 33m 1 Konstantin Shvachko 12/Dec/11 06:19

            People

            • Assignee:
              stack
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development