Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1804

Add a new block-volume device choosing policy that looks at free space

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.4-alpha
    • Fix Version/s: 2.1.0-beta
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      There is now a new option to have the DN take into account available disk space on each volume when choosing where to place a replica when performing an HDFS write. This can be enabled by setting the config "dfs.datanode.fsdataset.volume.choosing.policy" to the value "org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy".
      Show
      There is now a new option to have the DN take into account available disk space on each volume when choosing where to place a replica when performing an HDFS write. This can be enabled by setting the config "dfs.datanode.fsdataset.volume.choosing.policy" to the value "org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy".

      Description

      HDFS-1120 introduced pluggable block-volume choosing policies, but still carries the vanilla RoundRobin as its default.

      An additional implementation that also takes into consideration the free space remaining on the disk (or other params) should be a good addition as an alternative to vanilla RR.

        Attachments

        1. HDFS-1804.patch
          28 kB
          Aaron T. Myers
        2. HDFS-1804.patch
          29 kB
          Aaron T. Myers
        3. HDFS-1804.patch
          31 kB
          Aaron T. Myers
        4. HDFS-1804.patch
          31 kB
          Aaron T. Myers
        5. HDFS-1804.patch
          31 kB
          Aaron T. Myers

          Issue Links

            Activity

              People

              • Assignee:
                atm Aaron T. Myers
                Reporter:
                qwertymaniac Harsh J
              • Votes:
                0 Vote for this issue
                Watchers:
                30 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: