Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      [root@c1204 hadoop]# sudo -u hdfs hadoop fs -mkdir testQuotaFolder
      [root@c1204 hadoop]# sudo -u hdfs hadoop fs -count -q testQuotaFolder
      none inf none inf 1 0 0 testQuotaFolder
      [root@c1204 hadoop]# sudo -u hdfs hdfs dfsadmin -setQuota 1000 testQuotaFolder
      [root@c1204 hadoop]# sudo -u hdfs hdfs dfsadmin -setSpaceQuota 1000 testQuotaFolder
      [root@c1204 hadoop]# sudo -u hdfs hadoop fs -count -q testQuotaFolder
      1000 999 1000 1000 1 0 0 testQuotaFolder
      [root@c1204 hadoop]# sudo -u hdfs hdfs dfsadmin -setSpaceQuota 9223372036854775807 testQuotaFolder
      [root@c1204 hadoop]# sudo -u hdfs hdfs dfsadmin -setQuota 9223372036854775807 testQuotaFolder
      [root@c1204 hadoop]# sudo -u hdfs hadoop fs -count -q testQuotaFolder
      1000 999 1000 1000 1 0 0 testQuotaFolder
      [root@c1204 hadoop]# sudo -u hdfs hdfs dfsadmin -setQuota 9223372036854775806 testQuotaFolder
      [root@c1204 hadoop]# sudo -u hdfs hdfs dfsadmin -setSpaceQuota 9223372036854775806 testQuotaFolder
      [root@c1204 hadoop]# sudo -u hdfs hadoop fs -count -q testQuotaFolder
      9223372036854775806 9223372036854775805 9223372036854775806 9223372036854775806 1 0 0 testQuotaFolder

      The documentation at http://hadoop.apache.org/common/docs/r0.20.0/hdfs_quota_admin_guide.html says that the max quota size is Long.Max_Value which is = 9223372036854775807, but as you can see above, setting the quota to this value returns no error but doesn't actually change the quota value. Long.Max_Value - 1 works though.. so the documentation should be changed to reflect this behavior. Long.Max_Value + 1 returns an actual error (though it isn't the same as the negative number error):

      [root@c1204 hadoop]# sudo -u hdfs hdfs dfsadmin -setSpaceQuota 9223372036854775808 testQuotaFolder
      setSpaceQuota: For input string: "9223372036854775808"
      Usage: java DFSAdmin [-setSpaceQuota <quota> <dirname>...<dirname>]
      [root@c1204 hadoop]# sudo -u hdfs hdfs dfsadmin -setSpaceQuota -1 testQuotaFolder
      setSpaceQuota: Illegal option -1

        Activity

        Hide
        eli2 Eli Collins added a comment -

        Aside from updating the docs we should update TestQuota to cover the limit cases.

        Show
        eli2 Eli Collins added a comment - Aside from updating the docs we should update TestQuota to cover the limit cases.
        Hide
        sujay.rau Sujay Rau added a comment -

        Adds test that confirms current behavior of setting quotas at and around Long.Max_Value

        Show
        sujay.rau Sujay Rau added a comment - Adds test that confirms current behavior of setting quotas at and around Long.Max_Value
        Hide
        sujay.rau Sujay Rau added a comment -

        Adds test that confirms current behavior of setting quotas at and around Long.Max_Value.

        Show
        sujay.rau Sujay Rau added a comment - Adds test that confirms current behavior of setting quotas at and around Long.Max_Value.
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12531155/hdfs-3478.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2610//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2610//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531155/hdfs-3478.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2610//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2610//console This message is automatically generated.
        Hide
        eli2 Eli Collins added a comment -

        +1 looks great Sujay

        Style nits: these are copied from the other examples but fyi

        • the catch should go on the same line as the preceding bracket eg "} catch (IllegalArgumentException e) {" - you can write the following more simply {code}

          hasException = false;
          try

          { dfs.setQuota(testFolder, Long.MAX_VALUE + 1, 10); }

          catch (IllegalArgumentException e)

          { hasException = true; }

          assertTrue("Excpetion not thrown", hasException);

          as
          

          try

          { dfs.setQuota(testFolder, Long.MAX_VALUE + 1, 10); fail("Exception not thrown"); }

          catch (IllegalArgumentException e)

          { // Expected }
          Btw the current behavior comes from the following:
          
          

          // Long that indicates "leave current quota unchanged"
          public static final long QUOTA_DONT_SET = Long.MAX_VALUE;

          
          
        Show
        eli2 Eli Collins added a comment - +1 looks great Sujay Style nits: these are copied from the other examples but fyi the catch should go on the same line as the preceding bracket eg "} catch (IllegalArgumentException e) {" - you can write the following more simply {code} hasException = false; try { dfs.setQuota(testFolder, Long.MAX_VALUE + 1, 10); } catch (IllegalArgumentException e) { hasException = true; } assertTrue("Excpetion not thrown", hasException); as try { dfs.setQuota(testFolder, Long.MAX_VALUE + 1, 10); fail("Exception not thrown"); } catch (IllegalArgumentException e) { // Expected } Btw the current behavior comes from the following: // Long that indicates "leave current quota unchanged" public static final long QUOTA_DONT_SET = Long.MAX_VALUE;
        Hide
        eli2 Eli Collins added a comment -

        I've committed this. Thanks Sujay!

        Show
        eli2 Eli Collins added a comment - I've committed this. Thanks Sujay!
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2355 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2355/)
        HDFS-3478. Test quotas with Long.Max_Value. Contributed by Sujay Rau (Revision 1349634)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349634
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2355 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2355/ ) HDFS-3478 . Test quotas with Long.Max_Value. Contributed by Sujay Rau (Revision 1349634) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349634 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2428 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2428/)
        HDFS-3478. Test quotas with Long.Max_Value. Contributed by Sujay Rau (Revision 1349634)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349634
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2428 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2428/ ) HDFS-3478 . Test quotas with Long.Max_Value. Contributed by Sujay Rau (Revision 1349634) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349634 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2376 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2376/)
        HDFS-3478. Test quotas with Long.Max_Value. Contributed by Sujay Rau (Revision 1349634)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349634
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2376 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2376/ ) HDFS-3478 . Test quotas with Long.Max_Value. Contributed by Sujay Rau (Revision 1349634) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349634 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1075 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1075/)
        HDFS-3478. Test quotas with Long.Max_Value. Contributed by Sujay Rau (Revision 1349634)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349634
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1075 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1075/ ) HDFS-3478 . Test quotas with Long.Max_Value. Contributed by Sujay Rau (Revision 1349634) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349634 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1108 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1108/)
        HDFS-3478. Test quotas with Long.Max_Value. Contributed by Sujay Rau (Revision 1349634)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349634
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1108 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1108/ ) HDFS-3478 . Test quotas with Long.Max_Value. Contributed by Sujay Rau (Revision 1349634) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349634 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java

          People

          • Assignee:
            sujay.rau Sujay Rau
            Reporter:
            sujay.rau Sujay Rau
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development