Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4181

LeaseManager tries to double remove and prints extra messages

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.4, 2.0.2-alpha
    • Fix Version/s: 2.0.3-alpha, 0.23.5
    • Component/s: namenode
    • Labels:
      None

      Description

      When checkLeases() runs, internalReleaseLease() is called on the expired ones. When it returns true, the lease is already removed, yet it is tried again in checkLease(). This causes unnecessary ERROR messages to be logged. The line doing removing.add(p) should be removed.

      The internalReleaseLease() method logs a detailed message per call, so the extra INFO log message from checkLease() is redundant.

      The error message from removeLease() can be very big and needs to be cut down. When the namenode itself is holding a lot of leases for block recovery, hitting this error is very expensive. In one instance, slow block recovery caused the namenode to hold more than 42K leases. The one log line in this case was over 4 MB. The dump of data structure should be only enabled in debug mode.

      1. hdfs-4181-trunk.patch
        2 kB
        Kihwal Lee
      2. hdfs-4181-trunk.patch
        2 kB
        Kihwal Lee
      3. hdfs-4181-branch-0.23.patch
        2 kB
        Kihwal Lee
      4. hdfs-4181-branch-0.23.patch
        2 kB
        Kihwal Lee

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1257 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1257/)
        HDFS-4181. LeaseManager tries to double remove and prints extra messages (Kihwal Lee via daryn) (Revision 1408779)

        Result = FAILURE
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408779
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1257 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1257/ ) HDFS-4181 . LeaseManager tries to double remove and prints extra messages (Kihwal Lee via daryn) (Revision 1408779) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408779 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1226 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1226/)
        HDFS-4181. LeaseManager tries to double remove and prints extra messages (Kihwal Lee via daryn) (Revision 1408779)

        Result = FAILURE
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408779
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1226 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1226/ ) HDFS-4181 . LeaseManager tries to double remove and prints extra messages (Kihwal Lee via daryn) (Revision 1408779) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408779 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #435 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/435/)
        HDFS-4181. LeaseManager tries to double remove and prints extra messages (Kihwal Lee via daryn) (Revision 1408784)

        Result = SUCCESS
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408784
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #435 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/435/ ) HDFS-4181 . LeaseManager tries to double remove and prints extra messages (Kihwal Lee via daryn) (Revision 1408784) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408784 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #36 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/36/)
        HDFS-4181. LeaseManager tries to double remove and prints extra messages (Kihwal Lee via daryn) (Revision 1408779)

        Result = SUCCESS
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408779
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #36 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/36/ ) HDFS-4181 . LeaseManager tries to double remove and prints extra messages (Kihwal Lee via daryn) (Revision 1408779) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408779 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3008 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3008/)
        HDFS-4181. LeaseManager tries to double remove and prints extra messages (Kihwal Lee via daryn) (Revision 1408779)

        Result = SUCCESS
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408779
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3008 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3008/ ) HDFS-4181 . LeaseManager tries to double remove and prints extra messages (Kihwal Lee via daryn) (Revision 1408779) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408779 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        Hide
        Daryn Sharp added a comment -

        I've committed to trunk, branch 2 & 23. Thanks Kihwal!

        Show
        Daryn Sharp added a comment - I've committed to trunk, branch 2 & 23. Thanks Kihwal!
        Hide
        Daryn Sharp added a comment -

        +1 Pre-commit didn't run on the latest patches but since they essentially changed the log level, I'll commit anyway.

        Show
        Daryn Sharp added a comment - +1 Pre-commit didn't run on the latest patches but since they essentially changed the log level, I'll commit anyway.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12553203/hdfs-4181-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3489//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3489//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553203/hdfs-4181-trunk.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3489//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3489//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12553193/hdfs-4181-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3488//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3488//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553193/hdfs-4181-trunk.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3488//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3488//console This message is automatically generated.
        Hide
        Kihwal Lee added a comment -

        Thanks, Daryn. Here are the new patches.

        Show
        Kihwal Lee added a comment - Thanks, Daryn. Here are the new patches.
        Hide
        Daryn Sharp added a comment -

        It seems odd to check if debugging is enabled to issue an alternate error. I'd suggest changing:

        if (LOG.isDebugEnabled()) {
          LOG.error(src + " not found in lease.paths (=" + lease.paths + ")");
        } else {
          LOG.error(src + " not found in lease.paths.");
        }
        

        to

        LOG.error(src + " not found in lease.paths");
        if (LOG.isDebugEnabled()) {
          LOG.debug("lease.paths = (" + lease.paths + ")");
        }
        

        Otherwise it looks good if the pre-commit passes.

        Show
        Daryn Sharp added a comment - It seems odd to check if debugging is enabled to issue an alternate error. I'd suggest changing: if (LOG.isDebugEnabled()) { LOG.error(src + " not found in lease.paths (=" + lease.paths + ")" ); } else { LOG.error(src + " not found in lease.paths." ); } to LOG.error(src + " not found in lease.paths" ); if (LOG.isDebugEnabled()) { LOG.debug( "lease.paths = (" + lease.paths + ")" ); } Otherwise it looks good if the pre-commit passes.

          People

          • Assignee:
            Kihwal Lee
            Reporter:
            Kihwal Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development