Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1487

FSDirectory.removeBlock() should update diskspace count of the block owner node

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1, 0.20.2, 0.21.0
    • Fix Version/s: 0.22.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This bug leads a quota computation error. When calling abandonBlock(), the cached diskspace count of INodeDirectoryWithQuota is larger than the actual consumed space value.

      1. HDFS-1487.patch
        4 kB
        Zhong Wang
      2. HDFS-1487.patch
        1 kB
        Zhong Wang
      3. hdfs-1487-1.patch
        3 kB
        Eli Collins

        Issue Links

          Activity

          Hide
          Zhong Wang added a comment -

          Patch attached for review. Fix the bug and add a warning message when quota is violated. I have tested on my cluster and it works ok. I will add a test case soon, maybe in TestAbandonBlock.java. Any suggestions?

          Show
          Zhong Wang added a comment - Patch attached for review. Fix the bug and add a warning message when quota is violated. I have tested on my cluster and it works ok. I will add a test case soon, maybe in TestAbandonBlock.java. Any suggestions?
          Hide
          Zhong Wang added a comment -

          New patch attached with test case. Run Hudson.

          Show
          Zhong Wang added a comment - New patch attached with test case. Run Hudson.
          Hide
          Zhong Wang added a comment -

          Found another quota bug when testing the patch. Diskspace reduced twice when completing file. I think it is related to HDFS-1377.

          Linked with HDFS-1377. I will attach a new patch soon to fix this.

          Show
          Zhong Wang added a comment - Found another quota bug when testing the patch. Diskspace reduced twice when completing file. I think it is related to HDFS-1377 . Linked with HDFS-1377 . I will attach a new patch soon to fix this.
          Hide
          Eli Collins added a comment -

          +1 Good catch Zhong.

          Patch attached is rebased against HDFS-1507, I made the test junit4 style. Also removed the warning in FSDirectory; the patch I wrote for HDFS-1377 has an something which we can discuss there or in a separate jira.

          Show
          Eli Collins added a comment - +1 Good catch Zhong. Patch attached is rebased against HDFS-1507 , I made the test junit4 style. Also removed the warning in FSDirectory; the patch I wrote for HDFS-1377 has an something which we can discuss there or in a separate jira.
          Hide
          Eli Collins added a comment -

          I forgot to add that in my patch I pass true for checkQuota, since it should be checked by default (not that it would necessarily fire here since we're subtracting from the quota).

          Show
          Eli Collins added a comment - I forgot to add that in my patch I pass true for checkQuota, since it should be checked by default (not that it would necessarily fire here since we're subtracting from the quota).
          Hide
          Eli Collins added a comment -
               [exec] 
               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     -1 release audit.  The applied patch generated 103 release audit warnings (more than the trunk's current 1 warnings).
               [exec] 
               [exec]     +1 system test framework.  The patch passed system test framework compile.
               [exec] 
          

          Release audit is not due to this patch.

          Show
          Eli Collins added a comment - [exec] [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 103 release audit warnings (more than the trunk's current 1 warnings). [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] Release audit is not due to this patch.
          Hide
          Eli Collins added a comment -

          I've committed this. Will merge to 22 when the branch has been created. Thanks Zhong!

          Show
          Eli Collins added a comment - I've committed this. Will merge to 22 when the branch has been created. Thanks Zhong!
          Hide
          Eli Collins added a comment -

          Committed to branch 22.

          Show
          Eli Collins added a comment - Committed to branch 22.
          Hide
          Eli Collins added a comment -

          Patch for branch 20 attached.

          Show
          Eli Collins added a comment - Patch for branch 20 attached.

            People

            • Assignee:
              Zhong Wang
              Reporter:
              Zhong Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development