Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8147

Mover should not schedule two replicas to the same DN storage

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Scenario:
      1. Three DN cluster. For DNs storage type is like this.
      DN1 : DISK,ARCHIVE
      DN2 : DISK
      DN3 : DISK,ARCHIVE (All DNs are in same rack)
      2. One file with two replicas (In DN1 and DN2)
      3. Set file storage policy COLD
      4. Now execute Mover.

      Expected Result: File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE

      Actual Result: chooseTargetInSameNode() move D1:DISK block to D1:ARCHIVE, but in next iteration chooseTarget() for same rake is selecting again DN1:ARCHIVE for target where already same block exists.
      chooseTargetInSameNode() and chooseTarget() should not select the node as target where already same replica exists. The dispatcher will fail to move block as shown in the log below. Then, the Mover will try again in next iteration.

      Logs

      15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status message opReplaceBlock BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica FinalizedReplica, blk_1073741852_1028, FINALIZED
      

        Attachments

        1. HDFS-8147.patch
          2 kB
          Surendra Singh Lilhore
        2. HDFS-8147_4.patch
          4 kB
          Surendra Singh Lilhore
        3. HDFS-8147_3.patch
          5 kB
          Surendra Singh Lilhore
        4. HDFS-8147_2.patch
          3 kB
          Walter Su
        5. HDFS-8147_1.patch
          1 kB
          Surendra Singh Lilhore

          Issue Links

            Activity

              People

              • Assignee:
                surendrasingh Surendra Singh Lilhore
                Reporter:
                surendrasingh Surendra Singh Lilhore
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: