Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4495

Allow client-side lease renewal to be retried beyond soft-limit

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha, 0.23.5
    • Fix Version/s: 0.23.7, 2.1.0-beta
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      The current lease renewal code in DFSClient gives up after several retries, if the soft limit exceeds. This causes the client to abort. Without this self destruction behavior, lease renewal can be retried infinitely due to unrelated low-level issue and prevent DFSClient and associated Connection from getting garbage collected.

      However, giving up at passage of the soft limit has been shown to be too fragile; A long GC on namenode or transient network outage can make clients fail. We need the self-destruct behavior, but clients should be allowed to retry for longer period of time.

      1. hdfs-4495.patch
        3 kB
        Kihwal Lee
      2. hdfs-4495.b023.patch
        3 kB
        Kihwal Lee
      3. hdfs-4495.patch
        3 kB
        Kihwal Lee

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1344 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1344/)
        HDFS-4495. Allow client-side lease renewal to be retried beyond soft-limit. Contributed by Kihwal Lee (Revision 1445965)

        Result = SUCCESS
        kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445965
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1344 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1344/ ) HDFS-4495 . Allow client-side lease renewal to be retried beyond soft-limit. Contributed by Kihwal Lee (Revision 1445965) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445965 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1316 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1316/)
        HDFS-4495. Allow client-side lease renewal to be retried beyond soft-limit. Contributed by Kihwal Lee (Revision 1445965)

        Result = FAILURE
        kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445965
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1316 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1316/ ) HDFS-4495 . Allow client-side lease renewal to be retried beyond soft-limit. Contributed by Kihwal Lee (Revision 1445965) Result = FAILURE kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445965 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #525 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/525/)
        HDFS-4495. Allow client-side lease renewal to be retried beyond soft-limit. Contributed by Kihwal Lee (Revision 1445969)

        Result = SUCCESS
        kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445969
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #525 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/525/ ) HDFS-4495 . Allow client-side lease renewal to be retried beyond soft-limit. Contributed by Kihwal Lee (Revision 1445969) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445969 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #127 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/127/)
        HDFS-4495. Allow client-side lease renewal to be retried beyond soft-limit. Contributed by Kihwal Lee (Revision 1445965)

        Result = SUCCESS
        kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445965
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #127 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/127/ ) HDFS-4495 . Allow client-side lease renewal to be retried beyond soft-limit. Contributed by Kihwal Lee (Revision 1445965) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445965 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3358 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3358/)
        HDFS-4495. Allow client-side lease renewal to be retried beyond soft-limit. Contributed by Kihwal Lee (Revision 1445965)

        Result = SUCCESS
        kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445965
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3358 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3358/ ) HDFS-4495 . Allow client-side lease renewal to be retried beyond soft-limit. Contributed by Kihwal Lee (Revision 1445965) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445965 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        Hide
        Kihwal Lee added a comment -

        I've committed this to trunk, branch-2 and branch-0.23.

        Show
        Kihwal Lee added a comment - I've committed this to trunk, branch-2 and branch-0.23.
        Hide
        Daryn Sharp added a comment -

        +1

        Show
        Daryn Sharp added a comment - +1
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12569256/hdfs-4495.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3970//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3970//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12569256/hdfs-4495.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3970//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3970//console This message is automatically generated.
        Hide
        Kihwal Lee added a comment -

        I fixed the typo and posted the 0.23 version of patch.

        Show
        Kihwal Lee added a comment - I fixed the typo and posted the 0.23 version of patch.
        Hide
        Daryn Sharp added a comment -

        +1 Looks like a small reasonable change. Minor nit is test mentions "has beeb" instead of "has been". This should also go into 23, so please add an equivalent patch.

        Show
        Daryn Sharp added a comment - +1 Looks like a small reasonable change. Minor nit is test mentions "has beeb" instead of "has been". This should also go into 23, so please add an equivalent patch.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12569238/hdfs-4495.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3969//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3969//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12569238/hdfs-4495.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3969//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3969//console This message is automatically generated.
        Hide
        Kihwal Lee added a comment -

        Allowing clients to renew leases past the soft limit is acceptable. The lease soft limit is used by lease recovery and the way it works guarantees that no two writers are writing to the same file or block. When lease recovery is triggered by create() the file is recreated, so any activity on the old RBW block won't interfere with the new file. If append() causes a lease recovery, a block recovery is forced against the last RBW block and until it is done client gets RecoveryInProgressException. As a part of block recovery, existing writers are terminated and a new generation number is stamped on the block, so any existing client won't be able to write any longer. A client explicitly calling recoverLease() also works similarly.

        Show
        Kihwal Lee added a comment - Allowing clients to renew leases past the soft limit is acceptable. The lease soft limit is used by lease recovery and the way it works guarantees that no two writers are writing to the same file or block. When lease recovery is triggered by create() the file is recreated, so any activity on the old RBW block won't interfere with the new file. If append() causes a lease recovery, a block recovery is forced against the last RBW block and until it is done client gets RecoveryInProgressException. As a part of block recovery, existing writers are terminated and a new generation number is stamped on the block, so any existing client won't be able to write any longer. A client explicitly calling recoverLease() also works similarly.

          People

          • Assignee:
            Kihwal Lee
            Reporter:
            Kihwal Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development