Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1172

Blocks in newly completed files are considered under-replicated too quickly


    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
    • Hadoop Flags:


      I've seen this for a long time, and imagine it's a known issue, but couldn't find an existing JIRA. It often happens that we see the NN schedule replication on the last block of files very quickly after they're completed, before the other DNs in the pipeline have a chance to report the new block. This results in a lot of extra replication work on the cluster, as we replicate the block and then end up with multiple excess replicas which are very quickly deleted.


        1. HDFS-1172.patch
          3 kB
          Boris Shkolnik
        2. replicateBlocksFUC.patch
          4 kB
          Hairong Kuang
        3. replicateBlocksFUC1.patch
          8 kB
          Hairong Kuang
        4. replicateBlocksFUC1.patch
          8 kB
          Hairong Kuang
        5. hdfs-1172.txt
          20 kB
          Todd Lipcon
        6. hdfs-1172.txt
          21 kB
          Eli Collins
        7. HDFS-1172-150907.patch
          17 kB
          Walter Su
        8. HDFS-1172.008.patch
          14 kB
          Masatake Iwasaki
        9. HDFS-1172.009.patch
          15 kB
          Masatake Iwasaki
        10. HDFS-1172.010.patch
          16 kB
          Masatake Iwasaki
        11. HDFS-1172.011.patch
          16 kB
          Masatake Iwasaki
        12. HDFS-1172.012.patch
          16 kB
          Masatake Iwasaki
        13. HDFS-1172.013.patch
          13 kB
          Masatake Iwasaki
        14. HDFS-1172.014.patch
          13 kB
          Masatake Iwasaki
        15. HDFS-1172.014.patch
          13 kB
          Masatake Iwasaki

          Issue Links



              • Assignee:
                iwasakims Masatake Iwasaki
                tlipcon Todd Lipcon
              • Votes:
                1 Vote for this issue
                33 Start watching this issue


                • Created: