Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3941

Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 1.2.0, 1-win
    • namenode
    • None
    • Reviewed

    Description

      With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
      0. No more than one replica is placed within a NodeGroup
      1. First replica on the local node.
      2. Second and third replicas are within the same rack but remote rack with 1st replica.
      3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.

      Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.

      Attachments

        1. HDFS-3941.patch
          57 kB
          Junping Du
        2. HDFS-3941.005.patch
          53 kB
          Jing Zhao
        3. HDFS-3941.004.patch
          51 kB
          Jing Zhao
        4. HDFS-3941.003.patch
          49 kB
          Jing Zhao
        5. HDFS-3941.002.patch
          56 kB
          Jing Zhao

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            junping_du Junping Du
            junping_du Junping Du
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment