Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8147

Mover should not schedule two replicas to the same DN storage

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Scenario:
      1. Three DN cluster. For DNs storage type is like this.
      DN1 : DISK,ARCHIVE
      DN2 : DISK
      DN3 : DISK,ARCHIVE (All DNs are in same rack)
      2. One file with two replicas (In DN1 and DN2)
      3. Set file storage policy COLD
      4. Now execute Mover.

      Expected Result: File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE

      Actual Result: chooseTargetInSameNode() move D1:DISK block to D1:ARCHIVE, but in next iteration chooseTarget() for same rake is selecting again DN1:ARCHIVE for target where already same block exists.
      chooseTargetInSameNode() and chooseTarget() should not select the node as target where already same replica exists. The dispatcher will fail to move block as shown in the log below. Then, the Mover will try again in next iteration.

      Logs

      15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status message opReplaceBlock BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica FinalizedReplica, blk_1073741852_1028, FINALIZED
      

      Attachments

        1. HDFS-8147_1.patch
          1 kB
          Surendra Singh Lilhore
        2. HDFS-8147_2.patch
          3 kB
          Walter Su
        3. HDFS-8147_3.patch
          5 kB
          Surendra Singh Lilhore
        4. HDFS-8147_4.patch
          4 kB
          Surendra Singh Lilhore
        5. HDFS-8147.patch
          2 kB
          Surendra Singh Lilhore

        Issue Links

          Activity

            People

              surendralilhore Surendra Singh Lilhore
              surendralilhore Surendra Singh Lilhore
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: