Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9902

Support different values of dfs.datanode.du.reserved per storage type

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.7.2
    • 2.8.0, 2.7.4, 3.0.0-alpha1
    • datanode
    • None
    • Reviewed
    • Hide
      Reserved space can be configured independently for different storage types for clusters with heterogeneous storage. The 'dfs.datanode.du.reserved' property name can be suffixed with a storage types (i.e. one of ssd, disk, archival or ram_disk). e.g. reserved space for RAM_DISK storage can be configured using the property 'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is not configured then the value specified by 'dfs.datanode.du.reserved' will be used for all volumes.
      Show
      Reserved space can be configured independently for different storage types for clusters with heterogeneous storage. The 'dfs.datanode.du.reserved' property name can be suffixed with a storage types (i.e. one of ssd, disk, archival or ram_disk). e.g. reserved space for RAM_DISK storage can be configured using the property 'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is not configured then the value specified by 'dfs.datanode.du.reserved' will be used for all volumes.

    Description

      Now Hadoop support different storage type for DISK, SSD, ARCHIVE and RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
      The DISK size may be several TB and the RAM_DISK size may be only several tens of GB.
      The problem is that when I configure DISK and RAM_DISK (tmpfs) in the same DN, and I set dfs.datanode.du.reserved values 10GB, this will waste a lot of RAM_DISK size.
      Since the usage of RAM_DISK can be 100%, so I don't want dfs.datanode.du.reserved configured for DISK impacts the usage of tmpfs.
      So can we make a new configuration for RAM_DISK or just skip this configuration for RAM_DISK?

      Attachments

        1. HDFS-9902-branch-2.7.patch
          5 kB
          Brahma Reddy Battula
        2. HDFS-9902-05.patch
          6 kB
          Brahma Reddy Battula
        3. HDFS-9902-04.patch
          6 kB
          Brahma Reddy Battula
        4. HDFS-9902-03.patch
          6 kB
          Brahma Reddy Battula
        5. HDFS-9902-02.patch
          5 kB
          Brahma Reddy Battula
        6. HDFS-9902.patch
          2 kB
          Brahma Reddy Battula

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            brahmareddy Brahma Reddy Battula
            panyuxuan Pan Yuxuan
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment