Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1026

Quota checks fail for small files and quotas

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
    • Fix Version/s: None
    • Component/s: documentation, namenode
    • Labels:
    • Release Note:
      Quota check fails for small files and quotas are now being logged.

      Description

      If a directory has a quota less than blockSize * numReplicas then you can't add a file to it, even if the file size is less than the quota. This is because FSDirectory#addBlock updates the count assuming at least one block is written in full. We don't know how much of the block will be written when addBlock is called and supporting such small quotas is not important so perhaps we should document this and log an error message instead of making small (blockSize * numReplicas) quotas work.

      // check quota limits and updated space consumed
      updateCount(inodes, inodes.length-1, 0, fileINode.getPreferredBlockSize()*fileINode.getReplication(), true);
      

      You can reproduce with the following commands:

      $ dd if=/dev/zero of=temp bs=1000 count=64
      $ hadoop fs -mkdir /user/eli/dir
      $ hdfs dfsadmin -setSpaceQuota 191M /user/eli/dir
      $ hadoop fs -put temp /user/eli/dir  # Causes DSQuotaExceededException
      
      1. HDFS-1026.pacth
        1 kB
        Hızır Sefa İrken

        Issue Links

          Activity

          Hide
          Konstantin Shvachko added a comment -

          Document and log small quotas sounds right.

          Show
          Konstantin Shvachko added a comment - Document and log small quotas sounds right.
          Hide
          Konstantin Shvachko added a comment -

          Marking a blocker to make sure documentation is up to date.

          Show
          Konstantin Shvachko added a comment - Marking a blocker to make sure documentation is up to date.
          Hide
          Jim Plush added a comment -

          Were you thinking adding JavaDoc comments and updating the verifyQuota method in FSDirectoy.java? If not if you want to give a helpful pointer I'll be happy to make the fix.

          Show
          Jim Plush added a comment - Were you thinking adding JavaDoc comments and updating the verifyQuota method in FSDirectoy.java? If not if you want to give a helpful pointer I'll be happy to make the fix.
          Hide
          Eli Collins added a comment -

          Yes, add a comment in FSDirectory#addBlock and in the code that generates a quota violation exception LOG.warn that the quota check failed because the quota is smaller than block size * # replicas.

          Show
          Eli Collins added a comment - Yes, add a comment in FSDirectory#addBlock and in the code that generates a quota violation exception LOG.warn that the quota check failed because the quota is smaller than block size * # replicas.
          Hide
          Hızır Sefa İrken added a comment -

          From what I read, I prepared and attached a patch.

          Show
          Hızır Sefa İrken added a comment - From what I read, I prepared and attached a patch.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520273/HDFS-1026.pacth
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2212//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2212//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520273/HDFS-1026.pacth against trunk revision . +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2212//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2212//console This message is automatically generated.

            People

            • Assignee:
              Unassigned
              Reporter:
              Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development