Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11295

Check storage remaining instead of node remaining in BlockPlacementPolicyDefault.chooseReplicaToDelete()

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently in BlockPlacementPolicyDefault.chooseReplicaToDelete() the logic for choosing replica to delete is to pick the node with the least free space(node.getRemaining()), if all hearbeats are within the tolerable heartbeat interval.
      However, a node may have multiple storages and node.getRemaining() is a sum of the remainings of them, if free space of the storage with the block to be delete is low, free space of the node could still be high due to other storages of the node, finally the storage chosen may not be the storage with least free space.
      So using storage.getRemaining() to choose a storage with least free space for choosing replica to delete may be a better way to balance storage usage.

        Attachments

        1. HDFS-11295.001.patch
          4 kB
          Elek, Marton
        2. HDFS-11295.002.patch
          7 kB
          Elek, Marton
        3. HDFS-11295.003.patch
          7 kB
          Elek, Marton
        4. HDFS-11295.jpg
          123 kB
          Elek, Marton
        5. HDFS-11295.004.patch
          7 kB
          Elek, Marton

          Activity

            People

            • Assignee:
              elek Elek, Marton
              Reporter:
              Xiao Liang X. Liang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: