Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-761

Failure to process rename operation from edits log due to quota verification

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2, 0.21.0, 0.22.0
    • Fix Version/s: 0.20.2, 0.21.0
    • Component/s: namenode
    • Labels:
      None
    • Release Note:
      Corrected an error when checking quota policy that resulted in a failure to read the edits log, stopping the primary/secondary name node.

      Description

      When processing edits log, quota verification is not done and the used quota for directories is not updated. The update is done at the end of processing edits log. This rule is broken by change introduced in HDFS-677. This causes namenode from handling rename operation from edits log due to quota verification failure. Once this happens, namenode does not proceed edits log any further. This results in check point failure on backup node or secondary namenode. This also prevents namenode from coming up.

      1. hdfs-761.patch
        6 kB
        Suresh Srinivas
      2. hdfs-761.rel20.patch
        3 kB
        Suresh Srinivas
      3. hdfs-761.1.patch
        6 kB
        Suresh Srinivas
      4. hdfs-761.1.rel20.patch
        4 kB
        Suresh Srinivas
      5. hdfs-761.1.patch
        6 kB
        Suresh Srinivas
      6. hdfs-761.rel21.patch
        8 kB
        Suresh Srinivas

        Issue Links

          Activity

          Hide
          Suresh Srinivas added a comment -

          Attached patch has test that duplicates the problem and tests the fix.

          Show
          Suresh Srinivas added a comment - Attached patch has test that duplicates the problem and tests the fix.
          Hide
          Suresh Srinivas added a comment -

          Here is the exception thrown when the problem occurred:
          2009-11-09 21:54:24,671 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
          2009-11-09 21:54:24,671 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
          org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of ***** is exceeded:
          quota=13194139533312 diskspace consumed=40821.3g
          at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:149)
          at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1025)
          at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuotaForRename(FSDirectory.java:1051)
          at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedRenameTo(FSDirectory.java:409)
          at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:658)
          at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:992)
          at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:590)
          at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
          at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
          at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
          at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
          at java.lang.Thread.run(Thread.java:619)

          Show
          Suresh Srinivas added a comment - Here is the exception thrown when the problem occurred: 2009-11-09 21:54:24,671 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint: 2009-11-09 21:54:24,671 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of ***** is exceeded: quota=13194139533312 diskspace consumed=40821.3g at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:149) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1025) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuotaForRename(FSDirectory.java:1051) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedRenameTo(FSDirectory.java:409) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:658) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:992) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:590) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225) at java.lang.Thread.run(Thread.java:619)
          Hide
          Suresh Srinivas added a comment -

          Attaching a version of the patch for branch 0.20

          Show
          Suresh Srinivas added a comment - Attaching a version of the patch for branch 0.20
          Hide
          Hairong Kuang added a comment -

          Do you want to enable count updates during fsimage & fseditlogs merge?

          Show
          Hairong Kuang added a comment - Do you want to enable count updates during fsimage & fseditlogs merge?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12424524/hdfs-761.patch
          against trunk revision 834377.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/101/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/101/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/101/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/101/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424524/hdfs-761.patch against trunk revision 834377. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/101/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/101/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/101/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/101/console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          Not updating the counts seems to be the design intent. At the end of loading edits log, the counts are being updated in FSDirectory.updateCountForINodeWithQuota(). I would like to retain this and not change it.

          Show
          Suresh Srinivas added a comment - Not updating the counts seems to be the design intent. At the end of loading edits log, the counts are being updated in FSDirectory.updateCountForINodeWithQuota(). I would like to retain this and not change it.
          Hide
          Hairong Kuang added a comment -

          Then does it make sense to make the "ready" check in the beginning of verifyQuotaForRename?

          Show
          Hairong Kuang added a comment - Then does it make sense to make the "ready" check in the beginning of verifyQuotaForRename?
          Hide
          Suresh Srinivas added a comment -

          Added the check to verifyQuotaForRename() instead of relying upon the check in verifyQuota().

          Show
          Suresh Srinivas added a comment - Added the check to verifyQuotaForRename() instead of relying upon the check in verifyQuota().
          Hide
          Hairong Kuang added a comment -

          +1 The patch looks good.

          Show
          Hairong Kuang added a comment - +1 The patch looks good.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12424549/hdfs-761.1.rel20.patch
          against trunk revision 834377.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/102/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424549/hdfs-761.1.rel20.patch against trunk revision 834377. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/102/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12424571/hdfs-761.1.patch
          against trunk revision 834377.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/103/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/103/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/103/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/103/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424571/hdfs-761.1.patch against trunk revision 834377. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/103/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/103/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/103/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/103/console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #105 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/105/)
          . Fix failure to process rename operation from edits log due to quota verification. Contributed by Suresh Srinivas.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #105 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/105/ ) . Fix failure to process rename operation from edits log due to quota verification. Contributed by Suresh Srinivas.
          Hide
          Suresh Srinivas added a comment -

          Attaching a patch for branch 0.21

          Show
          Suresh Srinivas added a comment - Attaching a patch for branch 0.21
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 the 0.21 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 the 0.21 patch looks good.
          Hide
          Suresh Srinivas added a comment -

          The test failure related to TestBlockReport is unrelated to this change. I committed this change to trunk and branch 0.21.

          Show
          Suresh Srinivas added a comment - The test failure related to TestBlockReport is unrelated to this change. I committed this change to trunk and branch 0.21.
          Hide
          Suresh Srinivas added a comment -

          committed the patch to branch 0.20.

          Show
          Suresh Srinivas added a comment - committed the patch to branch 0.20.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #72 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/72/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #72 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/72/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #141 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/141/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #141 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/141/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #115 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/115/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #115 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/115/ )

            People

            • Assignee:
              Suresh Srinivas
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development