Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1948

Forward port 'hdfs-1520 lightweight namenode operation to trigger lease reccovery'

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Adds method to NameNode/ClientProtocol that allows for rude revoke of lease on current lease holder

      Description

      This issue is about forward porting from branch-0.20-append the little namenode api that facilitates stealing of a file's lease. The forward port would be an adaption of hdfs-1520 and its companion patches, hdfs-1555 and hdfs-1554, to suit the TRUNK.

      Intent is to get this fix into 0.22 time willing; i'll run a vote to get ok on getting it added to branch. HBase needs this facility.

      1. 1948-v4-minus_rpc_version_change.txt
        18 kB
        stack
      2. 1948-v4.txt
        18 kB
        stack
      3. 1948-v4.22.txt
        18 kB
        stack
      4. 1948-v3.txt
        1 kB
        stack
      5. 1948-v3.txt
        18 kB
        stack
      6. 1948-v2.txt
        18 kB
        stack
      7. 1948-part1.txt
        9 kB
        stack

        Issue Links

          Activity

          Hide
          Suresh Srinivas added a comment -

          HDFS-1948 forward ports HDFS-1554 to trunk

          Show
          Suresh Srinivas added a comment - HDFS-1948 forward ports HDFS-1554 to trunk
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #699 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/699/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #699 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/699/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #746 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/746/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #746 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/746/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #66 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/66/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #66 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/66/ )
          Hide
          stack added a comment -

          Applied patch to TRUNK and to branch-0.22 (HBase won't work on hadoop 0.22.0 if this patch is not present). Thanks for help and review Hairong.

          Show
          stack added a comment - Applied patch to TRUNK and to branch-0.22 (HBase won't work on hadoop 0.22.0 if this patch is not present). Thanks for help and review Hairong.
          Hide
          Hairong Kuang added a comment -

          +1 lgtm. Thanks, Stack!

          Show
          Hairong Kuang added a comment - +1 lgtm. Thanks, Stack!
          Hide
          stack added a comment -

          @Hairong I uploaded patch and above is what jenkins thinks of this version of the patch; its the same as v4 only minus the rpc version setting. You OK w/ my going ahead and committing this? (Thanks).

          Show
          stack added a comment - @Hairong I uploaded patch and above is what jenkins thinks of this version of the patch; its the same as v4 only minus the rpc version setting. You OK w/ my going ahead and committing this? (Thanks).
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481623/1948-v4-minus_rpc_version_change.txt
          against trunk revision 1132715.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/728//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/728//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/728//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481623/1948-v4-minus_rpc_version_change.txt against trunk revision 1132715. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/728//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/728//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/728//console This message is automatically generated.
          Hide
          stack added a comment -

          v4 of the patch for TRUNK minus rpc version increment.

          Show
          stack added a comment - v4 of the patch for TRUNK minus rpc version increment.
          Hide
          Hairong Kuang added a comment -

          Yes, please update the patch for the trunk.

          Show
          Hairong Kuang added a comment - Yes, please update the patch for the trunk.
          Hide
          stack added a comment -

          @Hairong Thanks for the review. OK with you if I commit the patches (I'll omit the bumping of rpc on commit)?

          Show
          stack added a comment - @Hairong Thanks for the review. OK with you if I commit the patches (I'll omit the bumping of rpc on commit)?
          Hide
          Hairong Kuang added a comment -

          Both patches look good! Stack, thanks for your time porting this feature.

          For the trunk, we already moved away from the version based RPC compatibility. Instead we take the method name base approach. Check HADOOP-6904. So there is no need to bump the version number. Instead dfsclient should use the method name to see if the server supports recoverLease or not.

          Show
          Hairong Kuang added a comment - Both patches look good! Stack, thanks for your time porting this feature. For the trunk, we already moved away from the version based RPC compatibility. Instead we take the method name base approach. Check HADOOP-6904 . So there is no need to bump the version number. Instead dfsclient should use the method name to see if the server supports recoverLease or not.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481434/1948-v4.22.txt
          against trunk revision 1131264.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/707//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481434/1948-v4.22.txt against trunk revision 1131264. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/707//console This message is automatically generated.
          Hide
          stack added a comment -

          I've been testing w/ this 0.22 patch and the lease grabbing seems to be working properly (We have explicit unit test in hbase for this – it passes – but did some cluster load killing of servers to test recovery and seems to do the right thing).

          Show
          stack added a comment - I've been testing w/ this 0.22 patch and the lease grabbing seems to be working properly (We have explicit unit test in hbase for this – it passes – but did some cluster load killing of servers to test recovery and seems to do the right thing).
          Hide
          stack added a comment -

          Here is a version for branch 0.22. Two minor changes only – the rpc version and how to get to leases.

          Show
          stack added a comment - Here is a version for branch 0.22. Two minor changes only – the rpc version and how to get to leases.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480626/1948-v4.txt
          against trunk revision 1128009.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/641//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/641//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/641//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480626/1948-v4.txt against trunk revision 1128009. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/641//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/641//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/641//console This message is automatically generated.
          Hide
          stack added a comment -

          Forgot to --no-prefix making patch

          Show
          stack added a comment - Forgot to --no-prefix making patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480625/1948-v3.txt
          against trunk revision 1128009.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/640//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480625/1948-v3.txt against trunk revision 1128009. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/640//console This message is automatically generated.
          Hide
          stack added a comment -

          Tests pass now (Had to figure the new-fangled user faking).

          Show
          stack added a comment - Tests pass now (Had to figure the new-fangled user faking).
          Hide
          stack added a comment -

          Here is full patch. Wrangling my test trying to get it to pass but putting this up meantime.

          Show
          stack added a comment - Here is full patch. Wrangling my test trying to get it to pass but putting this up meantime.
          Hide
          stack added a comment -

          Here is bulk of the port from branch-0.20-append of the recoverLease addition added by the following JIRAs.

          HDFS-1520. Lightweight NameNode operation recoverLease to trigger lease recovery. (Hairong Kuang via dhruba)
          HDFS-1555. Disallow pipelien recovery if a file is already being ease recovered (hairong)
          HDFS-1554. New semantics for recoverLease (hairong)

          Its not complete yet. The NameNode no longer has methods nextGenerationStamp (DatanodeProtocol) – they are present in 0.20.x but not in TRUNK – nor
          nextGenerationStampForBlock. I'm trying to figure what the equivalents are (Pointers welcome – thanks).

          Show
          stack added a comment - Here is bulk of the port from branch-0.20-append of the recoverLease addition added by the following JIRAs. HDFS-1520 . Lightweight NameNode operation recoverLease to trigger lease recovery. (Hairong Kuang via dhruba) HDFS-1555 . Disallow pipelien recovery if a file is already being ease recovered (hairong) HDFS-1554 . New semantics for recoverLease (hairong) Its not complete yet. The NameNode no longer has methods nextGenerationStamp (DatanodeProtocol) – they are present in 0.20.x but not in TRUNK – nor nextGenerationStampForBlock. I'm trying to figure what the equivalents are (Pointers welcome – thanks).

            People

            • Assignee:
              stack
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development