Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7584 Enable Quota Support for Storage Types
  3. HDFS-7701

Support reporting per storage type quota and usage with hadoop/hdfs shell

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      "hadoop fs -count -q" or "hdfs dfs -count -q" currently shows name space/disk space quota and remaining quota information. With HDFS-7584, we want to display per storage type quota and its remaining information as well.

      The current output format as shown below may not easily accomodate 6 more columns = 3 (existing storage types) * 2 (quota/remaining quota). With new storage types added in future, this will make the output even more crowded. There are also compatibility issues as we don't want to break any existing scripts monitoring hadoop fs -count -q output.

      $ hadoop fs -count -q -v /test
      QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME
      none inf 524288000 524266569 1 15 21431 /test

      Propose to add -t parameter to display ONLY the storage type quota information of the directory in the separately. This way, existing scripts will work as-is without using -t parameter.

      1) When -t is not followed by a specific storage type, quota and usage information for all storage types will be displayed.
      $ hadoop fs -count -q -t -h -v /test
      SSD_QUOTA REM_SSD_QUOTA DISK_QUOTA REM_DISK_QUOTA ARCHIVAL_QUOTA REM_ARCHIVAL_QUOTA PATHNAME
      512MB 256MB none inf none inf /test

      2) If -t is followed by a storage type, only the quota and remaining quota of the storage type is displayed.
      $ hadoop fs -count -q -t SSD -h -v /test

      SSD_QUOTA REM_SSD_QUOTA PATHNAME
      512 MB 256 MB /test

        Attachments

        1. HDFS-7701.01.patch
          17 kB
          Peter Shi
        2. HDFS-7701.02.patch
          19 kB
          Peter Shi
        3. HDFS-7701.03.patch
          20 kB
          Peter Shi
        4. HDFS-7701.04.patch
          21 kB
          Peter Shi
        5. HDFS-7701.05.patch
          21 kB
          Peter Shi
        6. HDFS-7701.06.patch
          21 kB
          Arpit Agarwal

          Issue Links

            Activity

              People

              • Assignee:
                shihaoliang Peter Shi
                Reporter:
                xyao Xiaoyu Yao
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: