Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-15

Rack replication policy can be violated for over replicated blocks

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.20.3
    • Fix Version/s: 0.20.3, 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      HDFS replicas placement strategy guarantees that the replicas of a block exist on at least two racks when its replication factor is greater than one. But fsck still reports that the replicas of some blocks end up on one rack.

      The cause of the problem is that decommission and corruption handling only check the block's replication factor but not the rack requirement. When an over-replicated block loses a replica due to decomission, corruption, or heartbeat lost, namenode does not take any action to guarantee that remaining replicas are on different racks.

      1. hdfs-15-b20-1.patch
        34 kB
        Eli Collins
      2. HDFS-15.6.patch
        19 kB
        Jitendra Nath Pandey
      3. HDFS-15.5.patch
        19 kB
        Jitendra Nath Pandey
      4. HDFS-15.4.patch
        18 kB
        Jitendra Nath Pandey
      5. HDFS-15.patch.3
        18 kB
        Jitendra Nath Pandey
      6. HDFS-15.patch.2
        17 kB
        Jitendra Nath Pandey
      7. HDFS-15.patch
        25 kB
        Jitendra Nath Pandey

        Issue Links

          Activity

          Hairong Kuang created issue -
          Hairong Kuang made changes -
          Field Original Value New Value
          Link This issue relates to HADOOP-4489 [ HADOOP-4489 ]
          Hairong Kuang made changes -
          Link This issue is related to HADOOP-4540 [ HADOOP-4540 ]
          Nigel Daley made changes -
          Fix Version/s 0.20.0 [ 12313438 ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] HDFS [ 12310942 ]
          Key HADOOP-4477 HDFS-15
          Component/s dfs [ 12310710 ]
          Jitendra Nath Pandey made changes -
          Assignee Hairong Kuang [ hairong ] Jitendra Nath Pandey [ jnp ]
          Jitendra Nath Pandey made changes -
          Attachment HDFS-15.patch [ 12417167 ]
          Jitendra Nath Pandey made changes -
          Attachment HDFS-15.patch.2 [ 12417311 ]
          Jitendra Nath Pandey made changes -
          Attachment HDFS-15.patch.3 [ 12417549 ]
          Jitendra Nath Pandey made changes -
          Attachment HDFS-15.4.patch [ 12417681 ]
          Jitendra Nath Pandey made changes -
          Attachment HDFS-15.5.patch [ 12417685 ]
          Jitendra Nath Pandey made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.21.0 [ 12314046 ]
          Jitendra Nath Pandey made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jitendra Nath Pandey made changes -
          Attachment HDFS-15.6.patch [ 12417944 ]
          Jitendra Nath Pandey made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jitendra Nath Pandey made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Eli Collins made changes -
          Fix Version/s 0.20.3 [ 12314814 ]
          Affects Version/s 0.20.3 [ 12314814 ]
          Eli Collins made changes -
          Attachment hdfs-15-b20-1.patch [ 12467143 ]
          Eli Collins made changes -
          Summary All replicas of a block end up on only 1 rack Rack replication policy can be violated for over replicated blocks
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to HDFS-1480 [ HDFS-1480 ]
          Eli Collins made changes -
          Link This issue is related to HDFS-1562 [ HDFS-1562 ]
          Todd Lipcon made changes -
          Link This issue is related to HDFS-3256 [ HDFS-3256 ]

            People

            • Assignee:
              Jitendra Nath Pandey
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development