Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9902

Support different values of dfs.datanode.du.reserved per storage type

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Reserved space can be configured independently for different storage types for clusters with heterogeneous storage. The 'dfs.datanode.du.reserved' property name can be suffixed with a storage types (i.e. one of ssd, disk, archival or ram_disk). e.g. reserved space for RAM_DISK storage can be configured using the property 'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is not configured then the value specified by 'dfs.datanode.du.reserved' will be used for all volumes.
      Show
      Reserved space can be configured independently for different storage types for clusters with heterogeneous storage. The 'dfs.datanode.du.reserved' property name can be suffixed with a storage types (i.e. one of ssd, disk, archival or ram_disk). e.g. reserved space for RAM_DISK storage can be configured using the property 'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is not configured then the value specified by 'dfs.datanode.du.reserved' will be used for all volumes.

      Description

      Now Hadoop support different storage type for DISK, SSD, ARCHIVE and RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
      The DISK size may be several TB and the RAM_DISK size may be only several tens of GB.
      The problem is that when I configure DISK and RAM_DISK (tmpfs) in the same DN, and I set dfs.datanode.du.reserved values 10GB, this will waste a lot of RAM_DISK size.
      Since the usage of RAM_DISK can be 100%, so I don't want dfs.datanode.du.reserved configured for DISK impacts the usage of tmpfs.
      So can we make a new configuration for RAM_DISK or just skip this configuration for RAM_DISK?

        Attachments

        1. HDFS-9902-branch-2.7.patch
          5 kB
          Brahma Reddy Battula
        2. HDFS-9902-05.patch
          6 kB
          Brahma Reddy Battula
        3. HDFS-9902-04.patch
          6 kB
          Brahma Reddy Battula
        4. HDFS-9902-03.patch
          6 kB
          Brahma Reddy Battula
        5. HDFS-9902-02.patch
          5 kB
          Brahma Reddy Battula
        6. HDFS-9902.patch
          2 kB
          Brahma Reddy Battula

          Issue Links

            Activity

              People

              • Assignee:
                brahmareddy Brahma Reddy Battula
                Reporter:
                panyuxuan Pan Yuxuan
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: