Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-76

Namespace quota exceeded message unclear

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      We seem to return one generic message when either a namespace or diskspace quota is exceeded. We should be clearer. The error should state which quota is exceeded and only report statistics on the exceeded quota.

      Further namespace quota is probably not clear to most users. We should probably have explanatory text such as: "you have exceeded the maximum allowed number of files". Maybe we should even rename the quota to number of files quota?

      Finally, the error below would be clearer if it read:

      Quota exceeded for directory '/user/ciyaliao'.

      By not using the word "directory" it is not clear in this case if the issue is with a user or a directory.

      ------ AN EXAMPLE -----
      > Hi,
      >
      > I encountered an error:
      >
      > put: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
      > /user/x is exceeded: namespace quota=200000 file count=200001,
      > diskspace quota=-1 diskspace=-44221467605496
      >
      > It apparently complains that dfs quota is exceeded. But I only use around
      > 200GB disk space. From the error message, what kind of quota did I actually
      > exceed? How do I check my usage of this type of quota?
      >
      > Thanks,
      > x

      1. HDFS-76.patch
        21 kB
        Raghu Angadi
      2. HADOOP-5862-1.patch
        21 kB
        Boris Shkolnik
      3. HADOOP-5862.patch
        21 kB
        Boris Shkolnik
      4. HADOOP-5862.patch
        21 kB
        Boris Shkolnik
      5. HADOOP-5862.patch
        19 kB
        Boris Shkolnik
      6. HADOOP-5862.patch
        17 kB
        Boris Shkolnik

        Issue Links

          Activity

          Hide
          Raghu Angadi added a comment -

          I just committed this to trunk. Thanks Boris!

          Show
          Raghu Angadi added a comment - I just committed this to trunk. Thanks Boris!
          Hide
          Raghu Angadi added a comment -

          patch for HDFS trunk.

          Show
          Raghu Angadi added a comment - patch for HDFS trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12411135/HADOOP-5862-1.patch
          against trunk revision 786278.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 14 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/537/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/537/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/537/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/537/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12411135/HADOOP-5862-1.patch against trunk revision 786278. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 14 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/537/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/537/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/537/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/537/console This message is automatically generated.
          Hide
          Boris Shkolnik added a comment -

          [exec] /home/sureshms/tools/buildtools/ant/apache-ant-1.7.1/bin/ant Dversion=PATCHHADOOP-5862-1.patch -Djava5.home=/usr/releng/tools/java/jdk1.5.0_06 -Dforrest.home=/home/ndaley/tools/forrest/latest -DHadoopPatchProcess= releaseaudit > ../scratch_dir//patchReleaseAuditWarnings.txt 2>&1
          [exec]
          [exec]
          [exec] There appear to be 499 release audit warnings before the patch and 499 release audit warnings after applying the patch.
          [exec]
          [exec]
          [exec]
          [exec]
          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 14 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          [exec]
          [exec]
          [exec]
          [exec]

          Show
          Boris Shkolnik added a comment - [exec] /home/sureshms/tools/buildtools/ant/apache-ant-1.7.1/bin/ant Dversion=PATCH HADOOP-5862 -1.patch -Djava5.home=/usr/releng/tools/java/jdk1.5.0_06 -Dforrest.home=/home/ndaley/tools/forrest/latest -DHadoopPatchProcess= releaseaudit > ../scratch_dir//patchReleaseAuditWarnings.txt 2>&1 [exec] [exec] [exec] There appear to be 499 release audit warnings before the patch and 499 release audit warnings after applying the patch. [exec] [exec] [exec] [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 14 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] [exec] [exec]
          Hide
          Boris Shkolnik added a comment -

          resynced with trunk

          Show
          Boris Shkolnik added a comment - resynced with trunk
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12410042/HADOOP-5862.patch
          against trunk revision 785065.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 14 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          -1 release audit. The applied patch generated 495 release audit warnings (more than the trunk's current 494 warnings).

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/504/testReport/
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/504/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/504/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/504/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/504/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12410042/HADOOP-5862.patch against trunk revision 785065. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 14 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 release audit. The applied patch generated 495 release audit warnings (more than the trunk's current 494 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/504/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/504/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/504/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/504/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/504/console This message is automatically generated.
          Hide
          Boris Shkolnik added a comment -

          Running test-patch on Friday, June 12, 09
          [exec]
          [exec]
          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 14 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          [exec]
          [exec]
          [exec]
          [exec]
          [exec] ======================================================================
          [exec] ======================================================================
          [exec] Finished build.
          [exec] ======================================================================
          [exec] ======================================================================
          [exec]

          Show
          Boris Shkolnik added a comment - Running test-patch on Friday, June 12, 09 [exec] [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 14 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== [exec]
          Hide
          Boris Shkolnik added a comment -

          synced with trunk to fix the patch

          Show
          Boris Shkolnik added a comment - synced with trunk to fix the patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12409707/HADOOP-5862.patch
          against trunk revision 781816.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 13 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/463/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12409707/HADOOP-5862.patch against trunk revision 781816. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 13 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/463/console This message is automatically generated.
          Hide
          Raghu Angadi added a comment -

          +1.

          Show
          Raghu Angadi added a comment - +1.
          Hide
          Boris Shkolnik added a comment -

          changed dfsadmin help

          Show
          Boris Shkolnik added a comment - changed dfsadmin help
          Hide
          Raghu Angadi added a comment -

          I think you forgot to fix 'hdfs dfsadmin -help' for these commands.

          Show
          Raghu Angadi added a comment - I think you forgot to fix 'hdfs dfsadmin -help' for these commands.
          Hide
          Boris Shkolnik added a comment -

          Implemented most of the comments.
          (left warning, but changed the wording).

          Show
          Boris Shkolnik added a comment - Implemented most of the comments. (left warning, but changed the wording).
          Hide
          Raghu Angadi added a comment -

          The patch looks good. A few comments apart from Nicholas comment above.

          1. Need to makes sure help for command line quota commands and documentation user guide reflect the new behaviour.
          2. In the Exception message, it might be better to replace "diskspace count=" with something like "diskspace consumed=". "count" might confuse users.
          3. Currently FSDirectory.updateCountForInodeWithQuota() warns when a directory exceeds quota. The warning text implies it is a software error (or corruption in the image) since it is not expected. The wording in the warning could be changed to be less severe.
          Show
          Raghu Angadi added a comment - The patch looks good. A few comments apart from Nicholas comment above. Need to makes sure help for command line quota commands and documentation user guide reflect the new behaviour. In the Exception message, it might be better to replace "diskspace count=" with something like "diskspace consumed=". "count" might confuse users. Currently FSDirectory.updateCountForInodeWithQuota() warns when a directory exceeds quota. The warning text implies it is a software error (or corruption in the image) since it is not expected. The wording in the warning could be changed to be less severe.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Boris, NameSpace quota not only counts files but also directories. It counts all "names".

          +      return "The NameSpace quota (maximum allowed number of files)" + ...
          
          Show
          Tsz Wo Nicholas Sze added a comment - Hi Boris, NameSpace quota not only counts files but also directories. It counts all "names". + return "The NameSpace quota (maximum allowed number of files)" + ...
          Hide
          Boris Shkolnik added a comment -

          implemented all of the above.

          Show
          Boris Shkolnik added a comment - implemented all of the above.
          Hide
          Jakob Homan added a comment - - edited

          For the diskspace quota, is it worth putting the numbers into MB, GB, etc for human consumption? On a large cluster with a large quota the formatted number would probably be more easily groked.

          Show
          Jakob Homan added a comment - - edited For the diskspace quota, is it worth putting the numbers into MB, GB, etc for human consumption? On a large cluster with a large quota the formatted number would probably be more easily groked.
          Hide
          Boris Shkolnik added a comment -

          Suggested wording for the namespace quota:
          The NameSpace quota (maximum allowed number of files) of directory /test is exceeded: quota=2 file count=3

          For disk quota:
          The DiskSpace quota of /test is exceeded: quota=45000 diskspace count=70656

          I will also modify TestQuota and TestHDFSCLI tests .

          Show
          Boris Shkolnik added a comment - Suggested wording for the namespace quota: The NameSpace quota (maximum allowed number of files) of directory /test is exceeded: quota=2 file count=3 For disk quota: The DiskSpace quota of /test is exceeded: quota=45000 diskspace count=70656 I will also modify TestQuota and TestHDFSCLI tests .
          Hide
          eric baldeschwieler added a comment -

          cool! thanks!

          Show
          eric baldeschwieler added a comment - cool! thanks!
          Hide
          Boris Shkolnik added a comment -

          Fixing together with HADOOP-5872.

          1. allowing to set quota even if the current usage goes over the quota.
          2. creating two separate exception for NameSpace and DiskSpace quota (both extending QuotaExceededException).

          Show
          Boris Shkolnik added a comment - Fixing together with HADOOP-5872 . 1. allowing to set quota even if the current usage goes over the quota. 2. creating two separate exception for NameSpace and DiskSpace quota (both extending QuotaExceededException).

            People

            • Assignee:
              Boris Shkolnik
              Reporter:
              eric baldeschwieler
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development