Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8131

Implement a space balanced block placement policy

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The default block placement policy will choose datanodes for new blocks randomly, which will result in unbalanced space used percent among datanodes after an cluster expansion. The old datanodes always are in high used percent of space and new added ones are in low percent.

      Through we can used the external balance tool to balance the space used rate, it will cost extra network IO and it's not easy to control the balance speed.

      An easy solution is to implement an balanced block placement policy which will choose low used percent datanodes for new blocks with a little high possibility. In a not long term, the used percent of datanodes will trend to be balanced.

      Suggestions and discussions are welcomed. Thanks

        Attachments

        1. balanced.png
          40 kB
          Liu Shaohui
        2. HDFS-8131.004.patch
          14 kB
          Liu Shaohui
        3. HDFS-8131.005.patch
          14 kB
          Liu Shaohui
        4. HDFS-8131.006.patch
          14 kB
          Liu Shaohui
        5. HDFS-8131-branch-2.7.patch
          15 kB
          Konstantin Shvachko
        6. HDFS-8131-v1.diff
          14 kB
          Liu Shaohui
        7. HDFS-8131-v2.diff
          14 kB
          Liu Shaohui
        8. HDFS-8131-v3.diff
          14 kB
          Liu Shaohui

          Issue Links

            Activity

              People

              • Assignee:
                liushaohui Liu Shaohui
                Reporter:
                liushaohui Liu Shaohui
              • Votes:
                0 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: