Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3685

Unbalanced replication target

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.17.0
    • 0.17.2
    • None
    • None
    • Reviewed

    Description

      In HADOOP-3633, namenode was assigning some datanodes to receive hundreds of blocks in a short period which caused datanodes to go out of memroy(threads).
      Most of them were from remote rack.

      Looking at the code,

          166           chooseLocalRack(results.get(1), excludedNodes, blocksize,
          167                           maxNodesPerRack, results);
      

      was sometimes not choosing the local rack of the writer(source).

      As a result, when a datanode goes down, other datanodes on the same rack were getting large number of blocks from remote racks.

      Attachments

        1. rereplicationPolicy1.patch
          10 kB
          Hairong Kuang
        2. rereplicationPolicy.patch
          10 kB
          Hairong Kuang

        Issue Links

          Activity

            People

              hairong Hairong Kuang
              knoguchi Koji Noguchi
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: