Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-253

Method to retrieve all quotas active on HDFS

    Details

    • Type: New Feature New Feature
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:

      Description

      Currently the only way to view quota information on an HDFS is via dfs -count -q, which is fine when an admin is examining a specific directory for quota status.

      It would also be good to do full HDFS quota audits, by pulling all HDFS quotas currently set on the system. This is especially important when trying to do capacity management ("OK, how much quota have we allotted so far?"). I think the only way to do this now is via lsr | count -q, which is pretty cumbersome.

      1. HDFS-253.patch
        44 kB
        Surendra Singh Lilhore
      2. HDFS-253-001.patch
        46 kB
        Surendra Singh Lilhore

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 21m 25s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 8m 2s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 24s The applied patch generated 3 new checkstyle issues (total was 194, now 194).
          -1 checkstyle 2m 53s The applied patch generated 1 new checkstyle issues (total was 126, now 127).
          +1 whitespace 0m 5s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 6m 2s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          -1 common tests 21m 31s Tests failed in hadoop-common.
          -1 hdfs tests 124m 42s Tests failed in hadoop-hdfs.
          +1 hdfs tests 0m 15s Tests passed in hadoop-hdfs-client.
              197m 6s  



          Reason Tests
          FindBugs module:hadoop-hdfs-client
          Failed unit tests hadoop.fs.TestLocalFsFCStatistics
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.TestAppendSnapshotTruncate
          Timed out tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12743896/HDFS-253-001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 81f3644
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11595/artifact/patchprocess/diffcheckstylehadoop-common.txt https://builds.apache.org/job/PreCommit-HDFS-Build/11595/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11595/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11595/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11595/artifact/patchprocess/testrun_hadoop-hdfs.txt
          hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/11595/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11595/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11595/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 21m 25s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 8m 2s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 24s The applied patch generated 3 new checkstyle issues (total was 194, now 194). -1 checkstyle 2m 53s The applied patch generated 1 new checkstyle issues (total was 126, now 127). +1 whitespace 0m 5s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 6m 2s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. -1 common tests 21m 31s Tests failed in hadoop-common. -1 hdfs tests 124m 42s Tests failed in hadoop-hdfs. +1 hdfs tests 0m 15s Tests passed in hadoop-hdfs-client.     197m 6s   Reason Tests FindBugs module:hadoop-hdfs-client Failed unit tests hadoop.fs.TestLocalFsFCStatistics   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.TestAppendSnapshotTruncate Timed out tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12743896/HDFS-253-001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 81f3644 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11595/artifact/patchprocess/diffcheckstylehadoop-common.txt https://builds.apache.org/job/PreCommit-HDFS-Build/11595/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11595/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11595/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11595/artifact/patchprocess/testrun_hadoop-hdfs.txt hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/11595/artifact/patchprocess/testrun_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11595/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11595/console This message was automatically generated.
          Hide
          Surendra Singh Lilhore added a comment -

          Fixed checkstyle and whitespace errors. Attached new path...

          Show
          Surendra Singh Lilhore added a comment - Fixed checkstyle and whitespace errors. Attached new path...
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 20m 45s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 23s The applied patch generated 22 new checkstyle issues (total was 194, now 214).
          -1 checkstyle 2m 53s The applied patch generated 2 new checkstyle issues (total was 126, now 128).
          -1 whitespace 0m 4s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 41s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 6m 6s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          -1 common tests 21m 44s Tests failed in hadoop-common.
          -1 hdfs tests 159m 37s Tests failed in hadoop-hdfs.
          +1 hdfs tests 0m 17s Tests passed in hadoop-hdfs-client.
              231m 22s  



          Reason Tests
          FindBugs module:hadoop-hdfs-client
          Failed unit tests hadoop.fs.TestHarFileSystem
            hadoop.fs.TestFilterFileSystem
            hadoop.hdfs.protocol.TestAnnotations



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12743712/HDFS-253.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / fc92d3e
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/diffcheckstylehadoop-common.txt https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/testrun_hadoop-hdfs.txt
          hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11582/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11582/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 45s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 23s The applied patch generated 22 new checkstyle issues (total was 194, now 214). -1 checkstyle 2m 53s The applied patch generated 2 new checkstyle issues (total was 126, now 128). -1 whitespace 0m 4s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 41s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 6m 6s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. -1 common tests 21m 44s Tests failed in hadoop-common. -1 hdfs tests 159m 37s Tests failed in hadoop-hdfs. +1 hdfs tests 0m 17s Tests passed in hadoop-hdfs-client.     231m 22s   Reason Tests FindBugs module:hadoop-hdfs-client Failed unit tests hadoop.fs.TestHarFileSystem   hadoop.fs.TestFilterFileSystem   hadoop.hdfs.protocol.TestAnnotations Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12743712/HDFS-253.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fc92d3e Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/diffcheckstylehadoop-common.txt https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/testrun_hadoop-hdfs.txt hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/11582/artifact/patchprocess/testrun_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11582/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11582/console This message was automatically generated.
          Hide
          Surendra Singh Lilhore added a comment -

          Attached initial patch , Please review...

          Show
          Surendra Singh Lilhore added a comment - Attached initial patch , Please review...
          Hide
          Surendra Singh Lilhore added a comment -

          Little change in above implementation.

          1. We already have Count command for getting the quota information. Same I added one new command listCount to get the content summary from HDFS.

           ./hdfd dfs -listCount [-q [-n | -s] ] [-h] [-r] [-v] <path> 
          
            List the directories with the quotas, directories, files and bytes count under
            the paths
            The -q option displays quotas.
            The -n option displays only name quota. It must be used with -q option.
            The -s option displays only space quota. It must be used with -q option.
            The -h option shows file sizes in human readable format.
            The -v option displays a header line.
          

          2. Added one new API listContentSummary() in ClientProtocol.java for retrieving ContentSummary from namenode , It will return DirectoriesContentSummary.

          3. Added one new API in FileSystem.java, it will return RemoteIterator<ContentSummary>.

          	public RemoteIterator<ContentSummary> listContentSummary(Path path, boolean recurcive
          
          Show
          Surendra Singh Lilhore added a comment - Little change in above implementation. 1. We already have Count command for getting the quota information. Same I added one new command listCount to get the content summary from HDFS. ./hdfd dfs -listCount [-q [-n | -s] ] [-h] [-r] [-v] <path> List the directories with the quotas, directories, files and bytes count under the paths The -q option displays quotas. The -n option displays only name quota. It must be used with -q option. The -s option displays only space quota. It must be used with -q option. The -h option shows file sizes in human readable format. The -v option displays a header line. 2. Added one new API listContentSummary() in ClientProtocol.java for retrieving ContentSummary from namenode , It will return DirectoriesContentSummary. 3. Added one new API in FileSystem.java, it will return RemoteIterator<ContentSummary>. public RemoteIterator<ContentSummary> listContentSummary(Path path, boolean recurcive
          Hide
          Surendra Singh Lilhore added a comment -

          Hi All,

          I am working on this jira and planning to do something like this.

          1. Adding one new admin command for list quota information from given path.

          	hdfs dfsadmin -listQuota -r -h <path>
          	

          This command will list the directories with quota information under the given path. Here "-r" is for recursively list the quota information.

          2. Adding one new API listQuotaInfo() in ClientProtocol.java for retrieving quota information from namenode , It will return DirectoriesQuotaInfo.

          	    public DirectoriesQuotaInfo listQuotaInfo(String path, int next) throws IOException;
                      
          	    public class DirectoriesQuotaInfo {  
          		private final ContentSummary[] contentSummarys;
          		private final int next;
          	    }
          	    

          3. One API in FileSystem.java, it will return RemoteIterator<ContentSummary>

          		public RemoteIterator<ContentSummary> listQuotaInfo(Path path) throws IOException;
          	    

          All the recursive logic I am implementing in admin command, listQuotaInfo() API only return the list of ContentSummary from the give path for the directories which has Quota.

          Is this correct implementation? Please give suggestion.

          Show
          Surendra Singh Lilhore added a comment - Hi All, I am working on this jira and planning to do something like this. 1. Adding one new admin command for list quota information from given path. hdfs dfsadmin -listQuota -r -h <path> This command will list the directories with quota information under the given path. Here "-r" is for recursively list the quota information. 2. Adding one new API listQuotaInfo() in ClientProtocol.java for retrieving quota information from namenode , It will return DirectoriesQuotaInfo . public DirectoriesQuotaInfo listQuotaInfo(String path, int next) throws IOException; public class DirectoriesQuotaInfo { private final ContentSummary[] contentSummarys; private final int next; } 3. One API in FileSystem.java, it will return RemoteIterator<ContentSummary> public RemoteIterator<ContentSummary> listQuotaInfo(Path path) throws IOException; All the recursive logic I am implementing in admin command, listQuotaInfo() API only return the list of ContentSummary from the give path for the directories which has Quota. Is this correct implementation? Please give suggestion.
          Hide
          Allen Wittenauer added a comment -

          One of my favorite JIRAs. Still open! I think I'll add the newbie tag.

          Show
          Allen Wittenauer added a comment - One of my favorite JIRAs. Still open! I think I'll add the newbie tag.

            People

            • Assignee:
              Surendra Singh Lilhore
              Reporter:
              Marco Nicosia
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:

                Development