Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1765

Block Replication should respect under-replication block priority

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 2.0.0-alpha, 0.23.7
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Currently under-replicated blocks are assigned different priorities depending on how many replicas a block has. However the replication monitor works on blocks in a round-robin fashion. So the newly added high priority blocks won't get replicated until all low-priority blocks are done. One example is that on decommissioning datanode WebUI we often observe that "blocks with only decommissioning replicas" do not get scheduled to replicate before other blocks, so risking data availability if the node is shutdown for repair before decommission completes.

        Attachments

        1. HDFS-1765.patch
          22 kB
          Uma Maheswara Rao G
        2. HDFS-1765.patch
          22 kB
          Uma Maheswara Rao G
        3. HDFS-1765.patch
          19 kB
          Uma Maheswara Rao G
        4. HDFS-1765.patch
          17 kB
          Uma Maheswara Rao G
        5. HDFS-1765.pdf
          560 kB
          Uma Maheswara Rao G
        6. underReplicatedQueue.pdf
          25 kB
          Haryadi Gunawi

          Issue Links

            Activity

              People

              • Assignee:
                umamaheswararao Uma Maheswara Rao G
                Reporter:
                hairong Hairong Kuang
              • Votes:
                0 Vote for this issue
                Watchers:
                19 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h