Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1804

Add a new block-volume device choosing policy that looks at free space

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.4-alpha
    • Fix Version/s: 2.1.0-beta
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      There is now a new option to have the DN take into account available disk space on each volume when choosing where to place a replica when performing an HDFS write. This can be enabled by setting the config "dfs.datanode.fsdataset.volume.choosing.policy" to the value "org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy".
      Show
      There is now a new option to have the DN take into account available disk space on each volume when choosing where to place a replica when performing an HDFS write. This can be enabled by setting the config "dfs.datanode.fsdataset.volume.choosing.policy" to the value "org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy".

      Description

      HDFS-1120 introduced pluggable block-volume choosing policies, but still carries the vanilla RoundRobin as its default.

      An additional implementation that also takes into consideration the free space remaining on the disk (or other params) should be a good addition as an alternative to vanilla RR.

      1. HDFS-1804.patch
        28 kB
        Aaron T. Myers
      2. HDFS-1804.patch
        29 kB
        Aaron T. Myers
      3. HDFS-1804.patch
        31 kB
        Aaron T. Myers
      4. HDFS-1804.patch
        31 kB
        Aaron T. Myers
      5. HDFS-1804.patch
        31 kB
        Aaron T. Myers

        Issue Links

          Activity

            People

            • Assignee:
              Aaron T. Myers
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development